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Review of Evidence: Arts Education Through the Lens of ESSA 


Executive Summary 


Purpose of the Report 


The Every Student Succeeds Act (ESSA), the 2015 reauthorization of the Elementary and 
Secondary Education Act, includes a diverse array of programs and funding streams that states, 
local educational agencies, and schools might leverage to support school improvement and 
student success. These programs and funding streams include approaches to teaching the arts 
and learning about the arts. Furthermore, ESSA contains provisions requiring or encouraging that 
educational agencies seeking to use federal funds available through the law for many of these 
approaches adopt evidence-based interventions. The inclusion of arts in ESSA shows that 
policymakers, not only arts advocacy groups and educators, view arts as an essential component in 
a well-rounded education. 


This report presents the results of a review of evidence about arts education interventions based 
on the evidence requirements in ESSA. According to ESSA, for an intervention to be evidence 
based, research or theoretical support for the intervention must fall within one of four evidence 
tiers. Evidence in Tiers I-IIl must “demonstrate a statistically significant effect on improving 
student outcomes or other relevant outcomes,” and the three tiers represent varying levels of rigor 
(i.e., “strong/moderate/promising evidence”). Tier IV evidence must “demonstrate a rationale” that 
an intervention is “likely to improve student outcomes or other relevant outcomes,” and it must be 
coupled with “ongoing efforts to examine the effects” of the intervention (ESSA, Title VIII, Section 
8101(21)(A)). School improvement activities funded through ESSA (Title |, section 1003) must 
include at least one intervention in one of the first three tiers. 


Definition of Arts Education 


The literature about arts education in prekindergarten through Grade 12 varies considerably in 
the use of terms to describe where and how students learn the knowledge and skills essential to 
an art type or discipline. For example, sometimes authors use the term “arts” to refer to all art 
types. When examining offerings in prekindergarten through Grade 12, the term “art” is often 
understood to refer to what we think of as “visual arts,” such as painting and drawing. We refer 
to this in our definition below. In the report we use the term “visual arts.” 


For the purposes of this review, the term “arts education” is used and is defined as follows: 


Arts lessons or classes offered in prekindergarten through Grade 12 that are 

(a) standards based and (b) taught by certified arts specialist teachers or teaching artists 
through (c) an explicit or implied sequential arts curriculum? in the (d) subjects of 
art/visual arts, media arts, music, dance, and drama/theater.? 


1 Sequential arts curriculum refers to courses or lessons designed to be coherent and taken in sequence in which 
students participate, starting at a beginning level and gradually moving toward a more advanced level of knowledge 
and skill. 

2 ESSA’s definition of a “well-rounded education” lists “the arts” and “music” as two distinct items. In this review, when 
the term “arts education” is used, music is understood to be one of the arts. Media arts can be considered a subject 
within arts; however, it is also often treated as its own art type. 
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Research Questions and Methods 
The evidence review addresses two research questions: 


1. Are there research studies on arts education interventions that meet the criteria for evidence 
as specified in ESSA? 


2. How large are the effects of arts education interventions on student outcomes?3 


To answer these two questions, we undertook five steps: 


1. Informed by published research reports, arts organization websites, and other resources, we 
developed a logic model describing the features of arts education and developed search 
criteria that would be used in two research database searches. 


2. We conducted two searches of educational research databases and clearinghouses to 
identify reports about the implementation and impact of arts education interventions. Both 
search efforts focused on research reported since 2000. The second search incorporated 
additional arts education terms to identify as many reports in scope as possible. The number 
of relevant reports found through these searches was 7,405. 


3. Our search was refined as we screened abstracts and then full-text documents with a focus on 
finding empirical studies with data about student outcomes in prekindergarten through Grade 
12. We screened out abstracts and texts deemed out of scope and identified 286 reports of 
studies for review by What Works Clearinghouse (WWC)-certified reviewers. 


4. We examined the reports that contained studies of arts education interventions and 
classified the studies according to definitions of the ESSA tiers of evidence (Tiers I-IV). 


5. Finally, to gain a fuller picture of the research findings on arts education interventions across 
all the well-designed studies, we recorded the magnitude of effects from 20 studies of arts 
education interventions and meta-analyzed those effect sizes. 


Box ES.1 provides definitions to clarify the distinctions among some of the key terms used in this 
evidence review report. 


3 Research question 2 originally expressed an interest in investigating the effects of arts education interventions 
particularly for students who are disadvantaged. The data reported in the studies included in this review, however, 
were not sufficient for this subgroup analysis. 
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Box ES.1. Definitions of Key Terms 


Arts education intervention: A specific approach, set of activities, strategy, or program addressing the 
instruction, content, or outcomes of the instruction in and/or participation in the study of the arts, including 
art/visual arts, media arts, music, dance, or drama/theater. 


Study: An empirical investigation of the effect of an arts education intervention on a particular sample and set 
of outcomes. Findings from a single study can appear in a single report or in multiple reports. 


Report: A written summary of a study, in the form of a journal article, a book or book chapter, a dissertation, a 
technical report, or a conference paper. A report may present findings from a single study or multiple studies. 


Report Highlights 


ESSA addresses the issue of arts education in several ways. It maintains an emphasis throughout 
its varied funding streams on ensuring that students have access to a “well-rounded education,” 
which, according to the law, can include “the arts” and “music” along with other subjects. Title IV of 
ESSA explicitly identifies programs in the arts and arts integration as allowable activities, and it 
provides for dedicated Assistance for Arts Education program. ESSA also offers funding for arts 
teaching and learning interventions that address the needs of specific student subgroups, such as 
economically disadvantaged students and English learners. 


ESSA includes more than 10 different funding opportunities that state educational agencies, 
local educational agencies, and schools can use to implement arts education interventions for 
students in prekindergarten through Grade 12. These funding opportunities can be used to 
support activities such as teacher professional development, school improvement efforts, supports 
for English learners, arts courses, instructional materials, and extended learning time programs. 
They can also be used to support arts-focused charter or magnet schools. 


Evidence of the effects of arts education interventions on student outcomes exists at three ESSA 
evidence tiers. However, most arts education interventions included in this review are supported by 
Tier IV evidence only. This review identified a total of 88 studies with evidence in one or more of 
the four ESSA evidence tiers. Eighteen of the studies met the evidence requirements for Tiers II 
and/or Ill. The other 70 studies provided evidence in Tier IV only; that is, these studies 
demonstrate a theory- or research-based rationale for why the interventions studied should 
improve student outcomes (e.g., academic achievement, art learning, social-emotional learning, 
and process abilities such as creativity or critical thinking), but they show no statistically 
significant positive effect of the interventions on those outcomes. 


The two art types with the largest number of studies with evidence in Tiers I-IV are music and 
visual arts. Fifty studies about music provided evidence aligned with ESSA tiers. The largest 
number of these studies, twenty-two, focused on art learning outcomes. Nineteen studies about 
visual arts provided evidence aligned with ESSA tiers. The largest number of these studies, ten, 
focused on process abilities as outcomes. 


According to the meta-analysis conducted as part of this evidence review, the average effect 
found in the 20 well-designed studies examined was moderate and statistically significant (effect 
size = 0.38), indicating that an average child would gain 15 percentile points in a relevant 
student outcome examined in this review (i.e., academic achievement, art learning, social- 
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emotional learning, and process abilities) as a result of participating in an arts education 
intervention. The 15-percentile-point increase would put the average effect of arts education 
interventions at the 75th percentile among the 70 interventions in mathematics, reading, and 
science reviewed by the WWC. It is important to note the effects of arts education interventions 
included in this review varied by art type as well as by outcome type. Further, for some art types, 
there was as few as one intervention with statistically significant positive effects. Educators 
should be cautious in making decisions based on these meta-analysis results considering that 
such a small number of studies could influence the magnitude of impact. 


Recommendations 


Based on this evidence review, we offer several recommendations for practitioners and policymakers 
as well as researchers who are likely to engage in selecting, implementing, and evaluating 
interventions focused on arts education. 


Recommendations for Practitioners and Policymakers 


Be thoughtful in selecting which ESSA funding program(s) to pursue to support a proposed 
arts education intervention. |mportant factors to consider include the types of activities that are 
required and allowable under the program(s), the amount and duration of funding available, and 
the level of evidence required. 


Critically assess the theoretical and empirical support behind a proposed arts education 
intervention. Adopting a conservative interpretation of ESSA’s evidence-based criteria might help 
promote interventions with a stronger likelinood of success. 


Before undertaking a search for arts education interventions that improve student outcomes, 
begin with a goal regarding the improvement focus as related to a particular art type and 
choose the intervention most likely to help reach that goal. As this evidence review notes, 
ESSA includes funding opportunities to use for arts education interventions and the literature 
contains many empirical reports of positive effects of such interventions on student outcomes. 
Still, practitioners and policymakers should be cautious when selecting interventions reporting 
positive evidence, as the studies providing such evidence may be limited in number, rigor, 
outcomes measured, and relevance to a particular context. Findings from reviews of studies by 
art type and by type of student outcomes may provide a more effective evidence base from 
which to consider implementing a particular arts education intervention. 


Recommendations for Researchers 


Research that is more rigorous is needed to provide stronger evidence for arts education. 
Researchers can help provide more Tier | evidence by using a randomized controlled trial study 
design, documenting the attrition of study participants, and providing sufficient details of 
analyses and findings in report appendices.4 


4|n a randomized controlled trial, study participants are randomly assigned to an intervention group that receives the 
intervention or to a control group that does not receive the intervention. 
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Work toward a clear definition of the arts experience, a precise logic model, and appropriate 
instruments to measure desired outcomes of arts education interventions to achieve a high 
level of evidence. |n Critical Links: Learning in the Arts and Student Academic and Social 
Development (Deasy, 2002), the reviewers of literature by art type consistently pointed out three 
challenges when examining the evidence about arts education interventions. The first is the 
existence of a description of the arts experience; the second is identifying the appropriate 
outcomes for the intervention; and the third is choosing the appropriate instruments to measure 
the outcomes. This evidence review has found similar omissions in studies conducted since 2000. 
Unfortunately, for most art types, except music, the literature continues to be sparse. The research 
literature on arts education interventions includes studies investigating the effects of arts 
education experiences on students’ scores on standardized tests and college entrance exams. 
Other studies examine very specific art techniques such as teaching music composition, pitch, or 
self-evaluation while learning to play an instrument. Still others examine neurological changes in 
children, as a result of interventions that may or may not be adaptable to the school setting. 


Additional research is needed to shed light on the effects of the use of arts education with 
diverse student populations in a range of settings. Researchers should plan their studies to 
include systematic comparisons of the effects of arts education on different student subgroups, 
such as students who are economically disadvantaged, English learners, and students with 
disabilities. Moreover, researchers should consider studying the effects of arts education among 
schools located in different settings (e.g., rural vs. suburban areas). 


Concluding Thoughts 


It may be timely to work toward a collaborative research agenda with emphasis on cross-cutting 
inquiry. |n each decade since the 1990s, reviews of the condition of research about the arts 
have been undertaken and research agendas for the arts have been articulated. In 1994, the 
Office of Educational Research and Improvement and the National Endowment for the Arts 
sponsored a review of research and needs (National Endowment for the Arts, 1994). Deasy’s 
(2002) Critical Links is another source of the condition of the literature and persisting research 
questions about arts education. Americans for the Arts and the American Educational Research 
Association sponsored the development of a research agenda (Arts Education Partnership, 
2004). The Dana Foundation developed an agenda for research on the brain and the arts 
(Asbury & Rich, 2008). The work of examining the field and identifying knowledge gaps as well as 
a research agenda has been taken on by organizations representing professionals and educators 
in various art fields, such as the National Association of Music Merchants, the National Dance 
Education Association (Bonbright & Faber, 2004; Bradley, Bonbright, & Dooling, 2013), and the 
National Art Education Foundation. Over the past two years, the National Endowment for the Arts 
invested in research labs around priority topics, such as the arts and social/emotional well-being, 
innovation, and creativity and learning. 


The identification of gaps from this evidence review may provide a jumping off point relevant for 
the future, as organizations consider a collaborative effort to examine the overarching questions 
that remain important to all the arts and could be pursued. 
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Chapter 1. What Do We Mean by “Arts Education?” 


Chapter Highlights 


> Arts education refers to arts lessons or classes offered in prekindergarten through Grade 
12 that are (a) standards based and (b) taught by certified arts specialist teachers or 
teaching artists through (c) an explicit or implied sequential arts curriculum in the 
(d) subjects of art/visual arts, media arts, music, dance, and drama/theater. 


> The practice of teaching about the arts in prekindergarten through Grade 12 has evolved 
with understanding about human development, technical and theoretical developments 
within each art discipline, and cultural and social influences, such as a shift in views 
about the importance of studying arts. 


> The theory and research-based logic model for arts education interventions suggests 
that arts education interventions may focus on intermediary outcomes, such as teacher 
capacity, that may affect student outcomes. Arts education interventions may solely 
focus on student outcomes, such as art mastery, attitudes about art or arts education, 
social-emotional learning, and “process” abilities, such as creativity and critical thinking. 


The 2015 reauthorization of the Elementary and Secondary Education Act as the Every Student 
Succeeds Act (ESSA) includes numerous federal funding opportunities that states, districts, and 
schools can use to support arts activities, strategies, and interventions designed to improve 
student learning. To help ensure the success of such interventions, the law requires or 
encourages that certain activities funded through ESSA be evidence based, and it outlines four 
evidence tiers for evaluating the level of evidence supporting a particular activity. 


Evidence in Tiers I-IIl must “demonstrate a statistically significant effect on improving student 
outcomes or other relevant outcomes,” and the three tiers represent varying levels of rigor (i.e., 
“strong/moderate/promising evidence”). Tier IV evidence must “demonstrate a rationale” that an 
intervention is “likely to improve student outcomes or other relevant outcomes,” and it must be 
coupled with “ongoing efforts to examine the effects” of the intervention (ESSA, Title VIII, Section 
8101(21)(A)). 


This report presents findings from an evidence review of recent research on arts education 
interventions through the lens of ESSA’s four-tiered evidence framework. The review addresses 
two research questions: 


1. Are there research studies on arts education interventions that meet the criteria for evidence 
as specified in ESSA? 


2. How large are the effects of arts education interventions on student outcomes?5 


5 This evidence review began with an additional goal to investigate the impact of arts education interventions 
particularly for students who are disadvantaged. The data reported in the studies included in this review, however, 
were not sufficient for this subgroup analysis. See Box 1.1 for definitions of key terms. 
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Box 1.1. Definitions of Key Terms 


Arts education intervention: A specific approach, set of activities, strategy, or program addressing the 
instruction, content, or outcomes of the instruction in and/or participation in the study of the arts, including 
art/visual arts, media arts, music, dance, or drama/theater. 


Study: An empirical investigation of the effect of an arts education intervention on a particular sample and set of 
outcomes. The findings from a single study can appear in a single report or in multiple reports. 


Report: A written summary of a study, in the form of a journal article, a book or book chapter, a dissertation, a 
technical report, or a conference paper. A report may present findings from a single study or from multiple studies. 


To address the above questions, the review team conducted a systematic search, screening, and 
review of reports of original studies published since 2000 that examined arts education 
interventions in prekindergarten through Grade 12 and their effects on student outcomes. In this 
chapter, we define the focus of the review, summarize the evolution of the practice of arts 
education, and describe characteristics of the arts education literature. 


Focus of the Evidence Review 


Prior to undertaking this evidence review, the review team conducted a pilot search of 
educational research databases and consulted a number of resources for clarity regarding the 
definition of arts education and the distinctions commonly made about art types or disciplines. 
Among the resources consulted were earlier, well-known research compendia (e.g., Deasy, 
2002). The National Arts Standards (httos://www.nationalartsstandards.org/) and websites for 
professional arts associations (e.g., National Dance Education Organization) were searched for 
sources of standards, art terms, and more recent statements about the status of research about 
arts education. In consultation with arts experts at The Wallace Foundation, we adopted the 
following definition of arts education for this evidence review: 


Arts lessons or classes offered in prekindergarten through Grade 12 that are (a) 
standards based and (b) taught by certified arts specialist teachers or teaching artists 
through (c) an explicit or implied sequential arts curriculum in the (d) subjects of 
art/visual arts, media arts, music, dance, and drama/theater.® 


This definition assumes the understanding that arts teaching and learning in prekindergarten 
through Grade 12 typically take place through instruction about concepts and skills related to a 
particular art form or discipline (e.g., art/visual arts, media arts, music, dance, or 
drama/theater). This is a distinguishing feature of arts education, as compared with arts 
integration, in that it takes place in a class or lesson devoted to the study of a particular art type. 
The definition also considers that the courses or lessons are designed to be coherent and taken 
in sequence, allowing participating students’ knowledge and skills to develop towards a more 
advanced level. 


6 ESSA’s definition of a “well-rounded education” lists “the arts” and “music” as two distinct items. In our review, we 
consider music as one of the “arts” when we use the term arts education. While “visual arts” is sometimes specified in 
reports, occasionally the term “art” is used to refer to types of visual arts, such as painting, drawing, photography. 
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Arts in Education: Evolving Practice and Access 


During the past 200 years, the content of arts education for each art type (i.e., what students 
study and produce) has expanded with input from professionals who practice that art and from 
teachers of the art (Szekely & Buckman, 2012). For example, the literature reviewers cited in 
Critical Links (Deasy, 2002) pointed to the evolution of the study of dance in public school, as 
having its origin in physical education classes (e.g., folk or square dance were often included in 
physical education). Similarly, reviewers of literature about drama education noted that prior to 
the existence of classes or clubs in theater, students were more likely to be engaged in the 
application of dramatic techniques in the study of literature (Deasy, 2002). 


Arts are taught in various ways in the United States, which can vary based on the level of the 
school, the type of art, or the philosophy, and educational theories. The instructional setting for the 
study of arts could be a whole class, an individual, or small-group lessons (e.g., individual or group 
lessons for learning a particular type of instrument). In prekindergarten through Grade 12, teachers 
who lead classes in the arts are likely to be certified arts teachers. Arts therapy interventions, arts 
programs in Summer camps, or afterschool arts programs were not within the scope of this 
evidence review. However, in the search for the literature, a number of studies of these out-of- 
scope interventions were located. For example, we identified studies of summer programs to 
prepare choral groups for upcoming competitions, studies of the use of arts to extend the 
educational opportunities for children or adults on the autism spectrum, and studies of a change in 
at-risk behavior in therapeutic or juvenile justice settings. 


One new trend in arts education is the use of various technologies, such as software or online 
platforms, to teach the arts (Clements & Sarama, 2003; Peppler, 2013). Another trend is to study 
media arts as a course of study. Media arts may be structured as the study of a specific type of 
technique (e.g., computer-generated arts, or graphic arts) or a broad area of study that encompasses 
many techniques for a specific career-focused use (e.g., animation for film and television, advertising, 
journalism, museum education, and composing). 


The question of access to arts education is one of the most common themes noted throughout the 
arts education literature. Evidence of an interest in showing the value of participating in arts 
education to students was noted recently when, at an Arts Education Partnership policy conference, 
state arts directors and arts educators highlighted efforts to identify a meaningful indicator about the 
arts in their ESSA-required state report cards. A recent report by the Brookings Institution called on 
arts educators to use available resources such as state and longitudinal databases to portray the 
extent of participation in the arts in their states (Kisida, Morrison, & Tuttle, 2017). 


Differences in participation in arts classes or lessons were highlighted in the sampling and focus 
used for the 2016 National Assessment of Educational Progress arts assessment (NAEP Arts), 
which was administered only to students in the eighth grade, highlighting the limited participation 
across secondary grades. Moreover, NAEP Arts focused solely on music and visual arts because 
only a small percentage of schools offered students opportunities to study dance and theater. 
NAEP Arts found that the percentage of eighth graders in the United States taking music and visual 
arts classes or lessons stayed about the same in 2016 as in 2008: 63% of the students were 
enrolled in a class in music, and 42% of the students took a course in visual arts in both years 
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(National Center for Education Statistics, 2017). Perhaps in recognition of the statistics about 
access to arts classes, reduced funding, and the national NAEP Arts results, there is renewed 
understanding of the value of studying arts as part of an elementary, middle school, or high school 
curriculum. The inclusion of arts in ESSA shows that policymakers, not only arts advocacy groups 
and educators, view arts as an essential component in a well-rounded education. 


The literature describing the history of arts in schools reflects advances in knowledge, political 
changes, and cultural changes, such as (a) an evolution in understanding how children learn; 
(b) increased value ascribed to education as a path to economic security; (c) changing 
expectations for student outcomes by the public and policymakers; (d) progressive, less 
formulaic approaches to curriculum and instruction; (e) shifting from an exclusive focus on the 
adult as artist to acceptance and value of the child as artist; (f) recognizing the importance of 
studying aesthetics and criticism; and (g) undertaking investigations into the relationship 
between different types of arts and other academic subjects (Fleming, 2012; National 
Endowment for the Arts, 1994; Szekely & Buckman, 2012). The resources from Fleming (2012), 
Szekely and Buckman (2012), and Stankiewicz, Amburgy, and Bolin (2004) are among many that 
point to some of the distinguishing features of various periods in the development of arts 
education in the United States. In Box 1.2, we summarize some of the notable events, people, 
and trends in arts education highlighted by these authors. 
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Box 1.2. Arts Education Practice: Evolving Definitions, Functions, and Key Players 


Pre-Colonial America 


> Families and societies passed down craft expertise. 
> Explorers and missionaries brought traditions from their countries to the southwest. 
> Artisans taught their craft to apprentices. 


Colonial America (1700s) 


> Tension existed between views of art’s purpose: What is “high art” versus practical or mechanic arts? 

> The study of the useful and/or the ornamental in art concerned primarily the education of White males since they 
were the only members of the population who owned property and were considered full citizens. 

> Historians note an evolving view of the value of studying art techniques, history, aesthetics, and criticism for 
individual improvement. 


Industrial America (1800s) 


> Educators maintained that the American schoolrooms contributed the tools, knowledge, and skills relevant to 
future jobs. 

> Art education was perceived as having special benefits for young women, and art and design schools were 
established as philanthropic enterprises in major cities. 

> Amajor initiative to prepare teachers for drawing instruction occurred and was driven by educator Walter Smith. 

> The artist professional community was developing. 

> Drawing books and cards written by working artists were incorporated in schools, and the belief in drawing as a 
means of communicating and improving knowledge became a focus, leading to its establishment as a course in 
school. 

> The Schoolroom Decoration and Picture-Study Movements focused on developing children’s character by seeing 
and appreciating religious and moral meaning in adult art. 

> Arthur Wesley Dow created the textbook Composition, explaining elements and principles of design. 

> Manual training reflected interest in “practical” forms of education and reflected views about gender, race, and 
youth at risk. 

> The National Education Association issued reports on American arts education. 


Development of Theories of Instruction and Curriculum: Influences from other disciplines (1900s) 


> The Child Study Movement emerged, which focused on understanding how children developed and learned and 
how to address problems. 

> Psychologists viewed art as one way of studying the emotional health and cognitive growth of the child and as a 
therapeutic activity. 

> Authors such as Viktor Lowenfeld and John Dewey produced theory-based works on arts education. 

> After the 1950s, with a national preoccupation on math and science, art curriculum alternatives were designed to 
support their place in formal education. 

> Getty Center for Education in the Arts seeded regional partnerships to implement balanced sequential arts 
instruction in the four areas of aesthetics, art criticism, art history, and art production. 

> The theme of school reform through standards and accountability emerged and was maintained through A Nation 
At Risk, Goals 2000, and advocacy organizations. 


Focus on Research and Dissemination (2000s) 


> Critical Links (Deasy, 2002) and Champions of Change (Fiske, 1999) noted challenges for research on the arts. 

> Arts in Education Model Development and Dissemination and Professional Development for Art Educators were 
established by the U.S. Department of Education. 

> National Core Arts Standards are updated and disseminated. 

> The National Center for Education Statistics (NCES) gathers data on the presence of arts education in public 
schools. 

> NAEP Arts is a source of data about what eighth graders know and do in music and arts. 

> Media technology becomes more present in schools at all levels, in software, as areas of study, and as the means 
to attract students to the arts. 


Sources include: Stankiewicz, Amburgy, & Bolin, 2004, Fleming, 2012, and Szekely & Buckman, 2012. 
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Research Literature About Arts in Education 


The drive to examine and improve the literature base about arts in education has intensified over 
the past 30 years. In each decade since the 1990s, reviews of the condition of research about 
the arts have been undertaken and research agendas for the arts have been articulated. The 
effect of this attention by collaborating national and professional organizations has moved the 
literature base from a primary focus on music and visual arts to an expanding literature base 
regarding dance and drama instruction and more rigorous research designs. 


In 1994, the Office of Educational Research and Improvement and the National Endowment for 
the Arts sponsored a review of research and needs (National Endowment for the Arts, 1994). 
Deasy’s (2002) Critical Links is another source of the condition of the literature and outstanding 
questions about arts education. Americans for the Arts and the American Educational Research 
Association sponsored the development of a research agenda on arts education (Arts Education 
Partnership, 2004). The Dana Foundation developed an agenda for research on the brain and 
the arts (Asbury & Rich, 2008). The Wallace Foundation has supported research about the 
quality of the arts experience (Seidel, Tishman, Winner, Hetland, & Palmer, 2009). 


Other reports have come forward from work in the field by the National Association of Music 
Merchants (Hodges & Luehrsen, 2010), the National Dance Education Association (Bonbright & 
Faber, 2004; Bradley, Bonbright, & Dooling, 2013), and the National Art Education Foundation. 
Over the past two years, the National Endowment for the Arts invested in research labs around 
priority topics, such as social/emotional well-being, the arts and creativity, innovation, and 
learning. 


The volume of published quantitative and qualitative studies of arts education has grown since 
2000. Overall, there are indications that new and experienced researchers acknowledge the 
interest in more rigorous design for research on arts education and are bringing research 
techniques to bear with this goal in mind. Some of the more recent research has been conducted 
by Doctoral students, many of whom began as certified teachers of art and music, from U.S. 
colleges and universities through their dissertations. The research conducted for a number of 
dissertations follows the groundbreaking approach by Catterall (2009), that is, an analysis of 
existing longitudinal databases in which investigators examine the magnitude of the relationship 
between participation in arts classes and experiences and success at various stages of adult life. 


Among other arts educators and researchers, there has been a dialogue regarding the 
measurement of outcomes. Authors have noted the shifts in research focus, describing a 
determination to measure the art experience itself and to avoid proxy indicators such as academic 
achievement to measure the value of studying an art type (Fleming, 2012; Winner & Hetland, 
2000). The literature contains studies of classroom conditions and technical skill development in a 
particular art form. Psychological attributes, such as motivation, creativity, and habits of mind, have 
also been investigated. The arena has expanded to consider the impact of participation in an arts- 
based learning experience for at-risk youth, for students with learning disabilities, and for students 
with economic disadvantages or second-language learning supports. 
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However, as highlighted in Deasy’s (2002) Critical Links, questions persist as challenges for 
researchers today: Are experiences in different disciplines enough alike so they can be lumped 
together in arts education? Why can’t we identify, define, and measure the collective multiple or 
integrated arts learning experience very well? What are the limitations of studies that measure 
arts learning using engagement or class participation as an indicator of improving outcomes? 
What methods are available to measure learning outcomes directly, whether these are creativity, 
self-concept, or performance? 


Logic Model for Arts Education Interventions 


In the literature, arts education interventions are theorized to impact student outcomes. The 
logic model as depicted in Figure 1.1 captures some of the key features, activities, and 
outcomes noted in the literature. 


Figure 1.1. Components, Processes, and Outcomes: A Logic Model of Arts Education 
Interventions 


Arts Education Interventions 
¢ Artdiscipline focus 


Intermediate Outcomes 


* Pedagogical approach (individual or group) 

+ Instructional level (beginner, intermediate, 
advanced) 

* Purpose (standards based, credit based) 

* Schedule of instructional time (e.g., in 
school, after school, out of school) 

* Target student characteristics 


Implementation Qualities 
Type of experience (traditional, 
innovative) 

Exposure (frequency and intensity) 
Enhancements (field trips, other) 
Emphasis (artistic, cognitive, 


Students’ attitudes toward arts 
education 
Improved teacher capacity 


* School or classroom climate 


Organizational participation 


+ Required teacher qualifications behavioral, etc.) Student Outcomes 
« Artcontent mastery and artistic 
product 
Be a eee eee Sete eee a ee * Academic achievement (nonart 


Contextual Factors 
National, state, and district standards 


' content mastery) 
Budgetary constraints 


Attitudes and dispositions 
Behavior outcomes 
Social-emotional outcomes 
Other cognitive outcomes 
(process abilities) 

* Success in life and work 


Resources 
Partnerships with arts or other organizations 


The box on the far left of the logic model represents arts education interventions as they were 
studied, and it describes some of the key features of these interventions. Arts education 
interventions, for example, typically include an emphasis on an art type or discipline (e.g., visual 
arts). Additional intervention features may include specific pedagogical approaches, instructional 
level, and credit-bearing status. Targeted student participation characteristics and required 
teacher qualifications may vary as well as scheduling. Arts education interventions may address 
the learning needs of different student populations, such as students with special learning needs 
or students of a particular grade level. 


The implementation qualities box identifies dimensions along which arts education interventions 
may vary. Arts education interventions can differ in instructional approach and content coverage: 
Some programs describe themselves as traditional, as in teacher-led lessons, and some 
interventions describe themselves as innovative, where students take a leadership role. The 
exposure of students to different interventions may vary in frequency and intensity. Some of the 
interventions include enhancements to regular classes such as field trips. Finally, approaches may 
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differ in their emphasis, that is, whether the focus is on creating art products, learning to think 
critically, or using studies in drama to address themes relevant to school behaviors, such as bullying. 


The implementation of arts education interventions may lead to improved intermediate 
outcomes that are expected ultimately to contribute to other key student outcomes. Examples of 
intermediate outcomes at the student level include students’ attitudes toward an art type or 
toward arts education in general. Some interventions focus on teacher practice, such as 
teachers’ ability to employ specialized instructional strategies, classroom interactions and 
behavior/motivation techniques, and assessment practices. Intermediate outcomes may include 
school-, parent-, or community-level measures such as school climate as well as the participation 
of partner organizations, such as museums. 


These intermediary outcomes may, in turn, affect key student outcomes such as art content mastery 
(e.g., art achievement) and the creation of art products (e.g., performance); academic achievement 
(i.e., nonart content mastery); attitudes and dispositions; behavior outcomes (e.g., attendance); 
social-emotional skills (e.g., self-confidence and self-awareness); other cognitive outcomes (i.e., 
process abilities such as creativity and critical thinking); and success in life and work. 


Contextual factors related to the circumstances in which an arts education intervention takes 
place can influence the intervention’s design as well as its implementation and outcomes. For 
example, a school district’s attention to recommendations from professional arts organizations or 
national standards regarding sequencing art classes may affect offerings in schools and 
expectations for student outcomes. Budgetary constraints may affect exploration of approaches 
and innovations such as the use of new technologies. The knowledge, skills, and dispositions of 
participating teachers might determine how much and what types of teacher support are included 
in an intervention’s design. Moreover, the availability of other resources that are important for 
sustaining an intervention over time (e€.g., community Support and school leadership buy-in) can 
play key roles in an intervention’s design and implementation. Similarly, features of the 
instructional setting (e.g., tyoe of physical space available and proximity to museums or performing 
arts centers) and the availability of instructional resources (e.g., necessary technology and art 
supplies) can shape how an arts education intervention is devised and carried out. Finally, it is 
possible that through the partnership with local or national arts support organizations, the arts 
education program in a school or throughout a district can be started and sustained. 


Organization of the Remainder of This Report 


In Chapter 2, we describe the funding opportunities available in ESSA for arts education 
interventions as well as the specific evidence requirements and guidelines for determining 
whether those interventions meet ESSA’s definition of evidence based. The findings from our 
evidence review are presented in Chapters 3 and 4. Chapter 3 identifies studies of arts 
education interventions that met the criteria for ESSA Evidence Tiers I-IV by art type. Chapter 4 
examines the magnitude of the effects of arts education interventions on student outcomes. A 
list of references follows Chapter 4. Appendix A provides detailed information about the review 
methods. Appendix B describes the criteria established by the U.S. Department of Education for 
each evidence tier for use in grant competitions that are administered directly by the 
Department. Finally, a large number of studies in this review of arts education interventions did 
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not provide evidence aligned with the requirements for Tiers I-IV. Rather than inserting multiple 
pages with this list in this report, we have created a separate listing that is available at the same 
location as the report. 
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Chapter 2. How Can the Every Student Succeeds 
Act Support Arts Education, and How Does It Define 
Evidence-Based Interventions? 


Chapter Highlights 


> ESSA offers over 10 different funding opportunities that state educational agencies, 
local educational agencies, and/or schools can use to support arts education. 


> Although ESSA’s definition of evidence-based interventions provides some criteria for 
evaluating the level of evidence supporting a particular intervention, it leaves room for 
interpretation. Interpretation for this evidence review was informed by prior evidence 
reviews and the expertise of colleagues at AIR and EducationCounsel. 


> This evidence review adopts the evidence criteria specified in the Non-Regulatory 
Guidance issued by the U.S. Department of Education in September 2016, which are 
stricter than the criteria specified in ESSA. 


In this chapter, we consider the decisions that practitioners and policymakers will make as they 
plan to use arts education interventions supported by funding under ESSA. First, we explore 
which funding programs within ESSA may lend themselves to supporting particular types of arts 
education interventions based on the programs’ stated purposes and allowable activities. We 
then unpack ESSA’s definition of “evidence based” interventions and explain how we applied 
that definition to our evidence review. 


Support for Arts Education Within ESSA 


ESSA addresses the issue of arts education in several important ways. For one, it broadens the 
subject area emphasis on mathematics, reading/English language arts, and science compared 
with its predecessor, the No Child Left Behind Act, to include a “well-rounded education.” 
Although the law does not dictate what subjects constitute a well-rounded education, it clarifies 
that a well-rounded education can include “the arts” and “music” along with other academic 
subject areas, such as history and foreign languages (ESSA Section 8101(52)). An increased 
focus on ensuring that students have equitable access to a well-rounded education runs 
throughout ESSA’s various titles and thus opens the door to using numerous ESSA funding 
programs to help finance arts education interventions. 


In addition to supporting the arts as part of a well-rounded education, ESSA features a funding 
stream devoted specifically to advancing arts education. Title IV, Part F of ESSA establishes a 
dedicated Assistance for Arts Education program (Sec. 4642 of ESSA, similar to the No Child Left 
Behind Act’s Arts in Education program), which supports a range of activities to promote education 
in music, dance, theater, media arts, and visual arts. State and local educational agencies can use 
Assistance for Arts Education funding for activities such as providing professional development to 
arts educators, developing instructional programs and materials in the arts, and expanding 
partnerships with museums, centers for the arts, and other arts organizations. 
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Last, ESSA provides funding opportunities to increase support for particular student subgroups 
such as students who are economically disadvantaged and English learners. Those opportunities 
can be used to fund interventions that help arts educators address those subgroups’ specialized 
learning needs. ESSA also requires states to set aside funds for school improvement activities in 
their lowest-performing schools, which could support the use of school turnaround strategies that 
incorporate sequential arts instruction, such as an arts-focused school improvement model or a 
school improvement model that includes arts instruction as part of a well-rounded education. 


To qualify for funding under a specific ESSA grant program, an arts education intervention must 
align with the grant program’s purpose and allowable activities. It also must satisfy any applicable 
evidence-based intervention requirements associated with that program. For certain ESSA programs, 
such as Title |, Section 1003: School Improvement, the law explicitly requires the use of evidence- 
based interventions for at least some of the activities allowable under that program. For other 
programs, the law indicates that evidence-based interventions must receive competitive preference 
in grant competitions, meaning that grant applications are not required to propose evidence-based 
interventions but must have a greater chance of being awarded if they do. Finally, some ESSA grant 
programs do not specify any evidence-based intervention requirements. Nevertheless, applicants 
may still choose to propose evidence-based interventions for these programs in the hopes, for 
example, that such interventions will be more likely to improve student outcomes. 


Table 2.1 presents a summary of ESSA funding opportunities that seem well suited for 
supporting arts education interventions, together with a summary of the applicable evidence- 
based intervention requirements for each opportunity listed. 


a 
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Table 2.1. Opportunities for Funding Under ESSA, Evidence Required for Funding, and Eligible ESSA Arts-Related Activities 


Opportunities for 
Supporting Arts Authorized Evidence Examples of Eligible Arts Education 
Education Description Eligible Entities Funding? Required> Activities 
Title I, Section Funds are intended to support _ LEAs with low- SEAs must At least one >» Arts-focused school turnaround 
1003: School improvement activities and performing schools _‘"eserve 7% of intervention must models that feature arts 
Improvement technical assistance for identified for their Title |, meet evidence instruction 

schools identified for comprehensive or Part A requirements of > Professional development for arts 

allocation. Tier |, Il, or Ill. educators in schools identified for 


comprehensive or targeted targeted support and 
Support and improvement improvement by their 
under their state accountability state’s accountability 
system.® system > Incorporating arts into a school 
improvement plan as a means of 
promoting a well-rounded 


comprehensive or targeted support 
and improvement 


education 
Title I, Section > Funds may be used to LEAs that reflect SEAs may No minimum > Arts courses that support schools’ 
1003A: Direct support academic geographic diversity "eserveupto — evidence threshold provision of a well-rounded 
coursework not otherwise 3% of their education 


within the state, with 
priority given to LEAs 


Student Services . 
available at the school, as 


well as personalized 


Title |, PartA > Advanced Placement, International 
allocation. Baccalaureate, and postsecondary 


learning, all aimed at 
improving college and 
career readiness and 
success. 


Examples include advanced 


courses, career and 
technical education 


courses, credit recovery or 


academic acceleration 


courses, and personalized 


learning approaches. 


American Institutes for Research 


serving the highest 
percentage of 
schools identified for 
comprehensive or 
targeted support and 
improvement 


level courses in the arts (and for 
low-income students, costs 
associated with exams that provide 
postsecondary credit in the arts) 


Courses that lead to industry- 
recognized credentials in the arts 
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Opportunities for 
Authorized 
Funding? 


Evidence 
Required> 


Supporting Arts 
Education 


Examples of Eligible Arts Education 


Eligible Entities Activities 


Title |, Part A: Basic 
Programs for 
Schoolwide and 
Targeted 
Assistance? 


Title Il, Part A: 
Supporting 
Effective 
Instruction4 


ee 


Description 


> Funds are intended to 
support schools and LEAs 
that serve high numbers or 
percentages of students 
from low-income families. 


> Schools operating Title | 
schoolwide programs can 
use their funds to improve 
the school’s entire 
educational program. 

> Schools operating targeted 
assistance programs must 
focus Title I-funded 
activities on students who 
are failing or most at risk of 
failing to meet state 
standards. 


Funds are intended to recruit 
and enhance the quality and 
effectiveness of current 
teachers, principals, and other 
school leaders. 


American Institutes for Research 


Schools where at 
least 40% of 
students come from 
low-income families 
(and other schools 
that receive a waiver 
from the state) 
qualify for Title | 
schoolwide 
programs; schools 
with smaller low- 
income populations 
can implement 
targeted assistance 
programs. 


LEAs 


$15 billion for 
FY 2017 to 
$16.2 billion 
for FY 2020 


Approximately 
$2.3 billion per 
year for FY 
2017 to 2020 


No minimum 
evidence threshold, 
except that any 
external providers 
selected to help 
schools implement 
their schoolwide or 
targeted 
assistance 
programs must 
have expertise in 
using evidence- 
based strategies 
(Tiers I-IV). 


Some specific uses 
of funds (é.g., 
professional 
development, 
induction, and 
mentoring) require 
Tier I, Il, II, or IV 
evidence, to the 
extent that the 
state determines 
such evidence is 
reasonably 
available.‘ 


> Whole-school improvement models 


or strategies that feature arts 
instruction 

Professional development to 
support arts educators 

Arts education programs or 
courses that support students’ 
access to a well-rounded 
education 


Arts programs that promote parent 
engagement 


Teacher professional development 
activities for arts educators 
Collaboration time for teaching 
artists and arts educators to plan 
lessons 

Time for teachers to develop 
curricula in the arts 

Financial incentives to recruit arts 
educators 

Financial incentives to help 
qualified individuals with an arts 
background become art educators 
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Opportunities for 

Supporting Arts Authorized Evidence Examples of Eligible Arts Education 
Education Description Eligible Entities Funding? Required> Activities* 

Title Ill, Part A: Supplemental funding provides LEAs or consortia of Approximately No minimum > Professional development to build 
English Language support for ELs in attaining multiple LEAs that $760 million evidence threshold arts educators’ capacity to meet 
Acquisition, English proficiency and serve sufficient for FY 2017 to the needs of ELs 

Language developing high levels of numbers of ELs $885 million > Acquiring digital resources that 
Enhancement, and academic achievement. for FY 2020 support ELs in arts classes 
Academic 


Achievement Act? 


Title IV, Part A: Funds provide support for: LEAs or consortia of | Approximately Some specific uses » Arts instructional programs to 

Student Support > providing all students with multiple LEAs $1.6 billion per of funds require provide students with a well- 

and Academic access to a well-rounded year Tier |, Il, Ill, or IV rounded education 

Enrichment Grants¢ education which may evidence, to the > Arts programs that promote 
include the arts, extent that the students’ physical and mental 

> improving school conditions state determines health, trauma recovery, or social- 

for student learning to such evidence is emotional skills (e.g., resilience, 
create a healthy and safe reasonably perseverance, self-management, 
school environment, and available.¢ empathy) 


> Professional development and 

to improve academic equipment to support the effective 
achievement and digital use of technology in delivering arts 
literacy. instruction, supporting 
collaboration among arts 
educators, and/or engaging with 
arts professionals outside of the 


> effectively using technology 


school 
Title IV, Part B: 21st Funds are intended to support LEAsor other public $1billionfor — Tier |, Il, Ill, or IV » Expanded learning time (e.g, 
Century Community academic enrichment activities or private nonprofit FY 2017 and — evidence, when before- or after-school programs) 
Learning Centers and other support services that entities, with priority $41.1 billion per deemed that provide arts instruction 
supplement regular school day _ given to those who year from FY appropriate > Summer school programs that 
requirements. serve students from 2018 to 2020 provide arts instruction 


high-poverty and low- 
performing schools 
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Opportunities for 
Supporting Arts Authorized Evidence Examples of Eligible Arts Education 
Education Description Eligible Entities Funding? Required> Activities*® 
Title IV, Part C: Start-up funds are provided for Charter management $270 to$300 No minimum > Planning activities for establishing 
Expanding creating new charter schools or organizations, with million per year evidence threshold or replicating charter schools with 
Opportunity replicating charter schools with _ priority given to those for FY 2017 to a specialized arts focus or 
Through Quality a track record of success. that serve high- 2020 curriculum 
Charter Schools poverty student > Professional development for 
populations teachers working in new charter 
schools to help them provide arts 
instruction 
Title IV, Part D: Start-up funds are provided for LEAs orconsortiaof $94 million for Competitive > Creating or replicating a magnet 
Magnet School establishing public schools or multiple LEAs that FY 2017 to preference is given school with a specialized arts focus 
Assistance education centers that offera  servestudents from $108 million for proposals with or curriculum 
specialized, theme-based diverse racial and for FY 2020 evidence-based > Professional development for 
instructional program designed socioeconomic activities (Tier I, Il, teachers working in a new magnet 
to attract and bring together backgrounds Ill, or IV). school to help them provide arts 
students from different racial instruction 
and socioeconomic 
backgrounds. 
Title IV, Part F: Funds are intended to support SEAs;LEAs; Bureau $70.5 million Includes program- >» +~Developing and evaluating a field- 
National Activities | the development, of Indian Education; for FY 2017 to specific evidence initiated arts teaching or learning 
Subpart 1— implementation, or scale-up of nonprofits; consortia $90.6 million requirements intervention 
Education innovations to improve high- of SEAs or LEAs; or _— for FY 2020 > Scaling up and rigorously 
Innovation and need students’ achievement partnerships between evaluating an arts teaching or 
Research and attainment, along with SEAs, LEAs, or learning intervention with positive 
rigorous evaluations of those Bureau of Education prior evaluation results to increase 
innovations. and nonprofits, its evidence base 
businesses, 
educational service 
agencies, or 
institutions of higher 
education 
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Opportunities for 
Supporting Arts 
Education 


Authorized 
Funding? 


Evidence 
Required> 


Examples of Eligible Arts Education 


Description Activities® 


Eligible Entities 


Title IV, Part F: 
National Activities 
Subpart 2— 
Community Support 
for School Success 


a 


Promise Neighborhoods 
grants support the provision 
of comprehensive, 
coordinated services to 
neighborhoods with high 
rates of poverty, multiple 
signs of distress (e.g., high 
rates of academic failure, 
obesity, incarceration), and 
low-performing schools. 


Full-Service Community 
Schools grants support the 
coordination and provision 
of pipeline services (a 
continuum of coordinated 
services from birth to 
postsecondary education 
and career attainment) in 
public elementary or 
secondary schools. 


American Institutes for Research 


Promise 
Neighborhoods: Non- 
profits, institutions of 
higher education, 
Indian tribes 


Full-Service 
Community Schools: 
LEAs working in 
partnership with one 
or more community- 
based organizations, 
nonprofit 
organizations, or 
other public or 
private entities 


Approximately 
$70.5 million 
per year for 
FYs 2017 and 
2018 and 
approximately 
$70 million per 
year for FYs 
2019 and 
2020 


Promise 
Neighborhoods: 
Application 
requirement 
includes support 
for evidence-based 
programs that 
assist students 
through transitions 
(e.g., pre-K—12 to 
postsecondary), 
and competitive 
preference is given 
for proposals with 
evidence-based 
activities (Tier |, Il, 
Ill, or IV). 
Full-Service 
Community 
Schools: 
Competitive 
preference is given 
for proposals with 
evidence-based 
activities (Tier |, Il, 
Ill, or IV). 


> 


> 


Arts initiatives that focus on 
improving students’ academic and 
developmental outcomes (e.g., 
school readiness, social-emotional 
skills, mental and physical health) 
Arts programs for early childhood 
students 

Extended learning time 
opportunities (e.g., before- or after- 
school or summer programs) that 
provide arts instruction 


Partnerships with community arts 
organizations to support arts 
initiatives 
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Opportunities for 
Supporting Arts Authorized Evidence Examples of Eligible Arts Education 
Education Description Eligible Entities Funding? Required® Activities*® 
Title IV, Part F: Funds include Assistance for > AnLEAora Approximately No minimum » Arts courses or programs 
National Activities | Arts Education Grants that consortium of $55 to $56 evidence threshold » Professional learning programs for 
Subpart 4—Awards _ support efforts to promote arts LEAs where at million per year art educators 
for Academic education, including arts leact 2070! for FY 2017 to > Development and use of 
Enrichment education for students who are Suet eis 2020 instructional materials and digital 
: from low-income 
disadvantaged and students -_ resources to support arts 
; eee families ; 
with disabilities. instruction 
> SEAs > Partnerships with arts 
> Institutions of organizations, museums, or 
higher education performing arts centers 
>» Museums or 
cultural 
institutions 


> Bureau of Indian 
Education 


> Other nonprofit or 
private 
organizations 


Note. EL = English learner. FY = fiscal year. LEA = local educational agency. SEA = state educational agency. 

aThe funding levels reported reflect the total amount of funding authorized in the law; they do not necessarily reflect the actual funding levels for each year, which are determined 
through the annual federal budget process. 

> This column reflects evidence requirements that are outlined in an ESSA statute. For grant competitions that are conducted directly by the U.S. Department of Education 
(rather than by SEAs), the Department might choose to add a competitive preference for evidence-based interventions even if not explicitly required under the law. 

¢ The activities listed are only examples of eligible arts activities for these programs as provided in statute, nonregulatory federal guidance, or otherwise; it is not an exhaustive list. 

d Indicates that states distribute funds to eligible entities based on a formula rather than on a competitive basis. For Title | 1OO3A Direct Student Services funds and Section 1003 School 
Improvement funds, states have the option of issuing subgrants to LEAs on either a formula or competitive basis. 

e Comprehensive support and improvement schools are schools among the lowest 5% of Title | schools across all required indicators within their state’s accountability system, schools 
that fail to graduate one third or more of their students, and Title | schools with chronically underperforming student subgroups. Targeted support and improvement schools are schools 
with one or more student subgroups that are consistently underperforming, as defined by the state, based on all required indicators within their state's accountability system. Schools 
with one or more subgroups that would perform, on their own, as poorly as the lowest 5% of Title | schools receive additional targeted support and improvement activities. 

fln some instances, ESSA allows states to waive evidence-based intervention requirements for specific uses of funds if the state determines that an evidence base is not reasonably 
available to apply to those requirements. 
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Identifying the Evidence Base of Arts Education Interventions 


Title VIII, Section 8101 of ESSA defines four tiers of evidence for evaluating the level of rigor in the 
research base for evidence-based interventions, as shown in Figure 2.1. Interventions supported by 
evidence in Tiers I-IIl must “demonstrate a statistically significant effect on improving student 
outcomes or other relevant outcomes,” and the three tiers represent varying levels of rigor (i.e., 
“strong, moderate, or promising evidence”). An intervention supported by Tier IV evidence must 
“demonstrate a rationale” that it is “likely to improve student outcomes or other relevant outcomes,” 
and it must be coupled with “ongoing efforts to examine the effects” of the intervention. 


Figure 2.1. “Evidence-Based” Intervention as Defined by ESSA 


WHAT IS AN “EVIDENCE-BASED” INTERVENTION? 
(from section 8101(21)(A) of the ESEA) 


“ the term “evidence-based.” when used with respect to a State, local educational agency, or school 
activity, means an activity, strategy, or intervention that — 


(i) demonstrates a statistically significant effect on improving student outcomes or other 


relevant outcomes based on — 
(¢3) strong evidence from at least one well-designed and well-implemented experimental 
study; 
¢5)) moderate evidence from at least one well-designed and well-implemented quasi- 
experimental study; or 
promising evidence from at least one well-designed and well-implemented 
correlational study with statistical controls for selection bias: or 
(Il) demonstrates a rationale based on high-quality research findings or positive 
evaluation that such activity, strategy. or intervention is likely to improve student 
outcomes or other relevant outcomes; and 


CD includes ongoing efforts to examine the effects of such activity. strategy. or 
intervention. 


As Table 2.1 in the previous section shows, under most federal programs that require or encourage the 
use of evidence-based interventions, interventions can satisfy ESSA’s definition of evidence-based if 
they meet the requirements for any of the four evidence tiers. However, the law requires that school 
improvement activities funded under Title |, Section 1003, must include at least one intervention that 
meets the evidence requirements for one of the first three (i.e., the most rigorous) tiers. 


Although ESSA’s definition of evidence-based interventions outlines general criteria for each evidence 
tier, it leaves room for states to specify more detailed criteria in areas in which the law is silent 
(Herman et al., 2016).” In this section, we outline the general approach that we took in interpreting 
and applying ESSA’s evidence-based intervention criteria for the purposes of this evidence review, 
which was informed by prior evidence reviews through the lens of ESSA and the expertise of colleagues 
knowledgeable about ESSA, the regulations, the guidance, and specific program requirements within 
ESSA. However, we recognize that guidance and interpretations of ESSA’s new requirements for 


7 The U.S. Department of Education has established its own criteria for each evidence tier for use in grant competitions that 
are administered directly by the Department (See Appendix B). 
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evidence-based interventions are still evolving as of this writing, and thus we encourage the reader to 
delve more deeply into available guidance and other resources on the evidence-based intervention 
criteria to make informed decisions. 


For this evidence review on arts education interventions, we adopted the stricter evidence criteria 
laid out in a September 2016 U.S. Department of Education document that provides nonregulatory 
guidance’ to states and school districts on how they might interpret the four evidence tiers (U.S. 
Department of Education, 2016). This document suggests that evidence tiers be defined using the 
following general criteria: 


> Strong evidence (Tier 1) comes from study reports that (a) show statistically significant positive 
intervention effects on relevant outcomes (without any statistically significant negative effects); 
(b) meet What Works Clearinghouse (WWC) evidence standards without reservations®; and (c) 
were conducted using a large, multisite sample (i.e., more than 350 students and more than a 
single school district). 


> Moderate evidence (Tier Il) comes from study reports that (a) show statistically significant 
positive intervention effects on relevant outcomes (without any statistically significant negative 
effects); (b) describe studies that meet WWC evidence standards with reservations; and (c) 
describe studies that were conducted using a large, multisite sample (i.e., more than 350 
students and more than a single school district). 


> Promising evidence (Tier III) comes from study reports that (a) show statistically significant 
positive intervention effects on relevant outcomes (without any statistically significant negative 
effects), and (b) describe correlational studies with statistical controls for selection bias. 


> Research-based rationale (Tier IV) evidence comes from study reports that (a) feature a well- 
specified logic model informed by research or evaluation, and (b) describe interventions that are 
undergoing additional research regarding their effects. 


As Herman et al. (2016) noted, the criterion for Tier IV evidence—a “research-based rationale”—is 
particularly challenging to apply because ambiguities in the definition of Tier IV evidence leave it open to 
broad or narrow interpretation. In their review of school leadership research, Herman et al. (2016) drew 
on information from the U.S. Department of Education’s September 2016 Non-Regulatory Guidance to 
interpret the first criterion for Tier IV evidence—a “well-specified logic model informed by research or 
evaluation”—to mean (a) a graphically presented logic model that includes key components of the 
intervention and outcomes, where (b) research or evaluation findings exist to support a connection 
between at least one intervention component and at least one relevant outcome (U.S. Department of 
Education, 2016, p. 9). 


Because the study reports collected for this review of arts education interventions vary considerably 
in how they present rationales or theories for the ways in which the interventions are expected to 


8 Non-Regulatory Guidance is information released by the U.S. Department of Education to assist SEAs and LEAs in 
implementing particular provisions of the law. It might include explanations, examples, and suggestions on how to 
implement the law’s provisions, but it does not require the use of specific activities or practices. 

2 Details about study designs that meet WWC evidence standards with or without reservations can be found in WWC’s 
standards handbook. This evidence review was guided by WWC’s Version 3.0 standards, which, together with the latest 
Version 4.0 standards, can be found at: https://ies.ed.gov/ncee/wwc/Handbooks. 
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influence student outcomes, we decided to employ a more inclusive approach to applying Tier IV 
evidence requirements. We considered studies that featured any type of rationale or theory of action 
linking features of the intervention to relevant outcomes as candidates for Tier IV evidence. We did 
not evaluate whether the studies met the second criterion for Tier IV evidence (i.e., that they describe 
an intervention that is undergoing additional research) as part of this review. 19 As we examined the 
studies that were candidates for Tier IV evidence, we noted that although an explicit logic model was 
absent in many of the studies, most included research references which the authors maintained 
were key to the intervention components and hypotheses guiding their investigations. This made 
sense to us, since the studies were published in peer-reviewed journals and/or were theses or 
dissertations. Conventionally, these kinds of publications often include a review of the literature 
related to the intervention investigated. In this report, unless there is an existing logic model in a 
study, we refer to the Tier IV evidence as being either a theory or research-based rationale. 


For this review, we Classified studies as providing no tier-aligned evidence if the following two 
conditions were met: (a) The study lacked statistically significant findings or used a research design 
other than those eligible for providing evidence in Tiers I-III and (b) the report describing the study 
lacked any type of rationale or logic model for the intervention studied. More details about the 
methodology used to identify studies on arts education interventions and classify the evidence from 
those studies according to the ESSA evidence tiers are provided in Appendix A. 


10 Due to our more inclusive approach regarding studies of interventions with Tier IV evidence, the interventions we 
identified as providing Tier IV evidence may require additional scrutiny to determine whether they satisfy the Department’s 
criteria for Tier IV evidence. 
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Chapter 3. What Evidence Exists Linking Arts Education 
Interventions With Improved Student Outcomes? 


Chapter Highlights 


> Evidence of the effects of arts education interventions on student outcomes exists at 
three of ESSA’s evidence tiers. 


Three studies provided evidence at Tier Il and 15 studies provided Tier IIl evidence 
regarding the type of arts education interventions within the scope of this evidence 
review. 


Seventy studies provided evidence at Tier IV for arts education interventions. 


No studies of arts education interventions provided Tier | evidence. 


The findings from the evidence review are divided into two chapters. Chapter 3 presents findings for 
the first research question (repeated below), and Chapter 4 presents the meta-analytic findings for 
the second research question. 


Research question 1: Are there research studies on arts education interventions that meet 
the criteria for evidence as specified in ESSA? 


To address this question, we classified the evidence from each relevant study on arts education 
interventions based on whether the study and its findings met the criteria for ESSA’s Evidence Tiers 
I-IV. Figure 3.1 summarizes the number of studies providing evidence at one of the four tiers, overall 
and by type of art and type of outcome. The bars in each cell indicate the numbers of studies 
providing evidence at Tiers Il, Ill, and IV respectively. Empty cells (shaded grey) indicate that no study 
reviewed provides evidence aligned with ESSA tiers for that particular art type and outcome domain. 
The details about each study, its findings, and the reason for its classification are provided in this 
chapter in sections devoted to the art type. 
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Figure 3.1. Evidence of the Effects of Arts Education Interventions on Student Outcomes: Number of Studies That Meet the Criteria for ESSA 
Evidence Tiers, by Art Type, Outcome Domain, and Tier of Evidence 


Outcome Domain 


Social-emotional Attitude Process abilities Other outcomes All outcomes 


Visual art 


Multiple art 


All art types 


mum Tier || mums Tier Ill meee Tier IV 


Note. Shaded cells indicate that no study reviewed provides evidence aligned with ESSA tiers for that particular art type and outcome domain. 
Source: Authors’ analysis of findings from literature review on arts education interventions. 
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As Figure 3.1 shows, evidence of the effects of arts education interventions on student outcomes 
exists at ESSA evidence Tiers Il, Ill, and IV. No interventions were supported by ESSA Tier | 
evidence. A total of 88 studies provided evidence meeting ESSA evidence criteria for 87 arts 
education interventions1: three studies with Tier Il evidence, 15 studies with Tier Ill evidence, 
and 70 studies with Tier IV evidence. The number of studies of interventions in the arts of music 
(50 studies) and visual arts (19 studies) far exceeds that for the arts of drama (eight studies) 
and dance (four studies). 


Most studies of arts education interventions included in this review provided Tier IV evidence only. 
Reports providing this type of evidence lack empirical results from rigorous studies, but they do 
contain a theory- or research-based rationale for why the intervention should improve student 
outcomes. Two studies provided evidence that could be classified at both Tier II and Tier Ill; 10 studies 
provided evidence that could be classified at both Tier Ill and Tier IV. These studies are represented 
just once—in the count for the higher level of evidence—within each cell of Figure 3.1. Nineteen studies 
reported evidence in multiple outcome domains that could be classified at Tiers Il, Ill, or IV. 


In the remainder of this chapter, we present evidence supporting arts education interventions by type 
of art. There are two tables for each art type: One table lists studies of interventions with evidence at 
Tiers Il and Ill, and the second table lists studies of interventions with evidence at Tier IV. Because 
the Tier lV table includes many more studies, it lists the studies of interventions alphabetically by 
leading author within each grade span for ease of locating the studies within the reference list. 


Evidence in Studies of Visual Arts Interventions 


Our review found 19 studies of visual arts interventions that provided evidence at Tiers Ill and IV. 
One study produced evidence aligned with Tier Ill (Table 3.1), and the other 18 studies provided 
Tier IV evidence (Table 3.2). 


Catterall and Peppler (2007) examined the effects of two visual arts programs serving inner-city 
9-year-olds in two U.S cities. The Inner City Arts (ICA) is an institution in downtown Los Angeles 
serving inner-city schools. The Center of Contemporary Arts (COCA) is a visual arts program that 
took place in schools in the public housing projects in St. Louis, Missouri. Both ICA and COCA 
featured, as per the authors, rich, sustained visual arts instruction. The study found that 
students who participated in the two programs made significantly greater gains than comparison 
students in self-efficacy beliefs and in originality (i.e., a process ability) as measured by a 
children’s version of a standard creativity test. The authors asserted that these effects were 
attributed to children’s engagement in art and to diverse positive interactions with peers and 
expert instructors surrounding the work. 


The 18 studies that provided Tier IV evidence addressed a range of outcomes, including art 
learning (e.g., quality of visual art work), social-emotional learning (e.g., self-efficacy, self-esteem, 
self-confidence, social responsibility), process abilities (e.g., critical thinking), and attitudes (e.g., 
attitude toward and interest in art). No studies of visual arts interventions provided evidence 
aligned with Tier IV for academic achievement outcomes. 


11 Two of the studies examined the same intervention. 
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Table 3.1. Evidence Review Results: Studies of Visual Arts Interventions With Tiers II and III Evidence 


Name of 
Intervention 


Geographic 
Location 


Intervention Components 


Outcomes for 
Which Positive 
Effects Were 
Found 


Characteristics of 
Study Samples 


Evidence 


Tier Abstract Link 


3 Inner City Arts > Los Angeles, 
(ICA): Los CA 
Angeles > St. Louis, MO 
Center of 
Contemporary 
Arts (COCA): St 
Louis 


a 


American Institutes for Research 


> Teaching artists, arts 
specialists 

> Sustained instruction 
(ICA—90 minutes twice 
per week for 20 weeks; 
COCA—60 minutes once 
per week for 30 weeks) 


> Social- Tier Ill > ICA-97% of Catterall & https://eric.ed.gov/?i 
emotional students qualifying Peppler(2007) d=EJ779868 
learning: Self- for free- or reduced- 
efficacy price lunch, 97% 

> Process Hispanic 
abilities: >» COCA—99% 
Originality students qualifying 


for subsidized 
meals, 100% African 
American 
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Table 3.2. Evidence Review Results: Studies of Visual Arts Interventions With Tier IV Evidence 


Grade | Name of Intervention | Geographic Intervention Abstract Link 
Level Location Components 


Elementary Grades (Prekindergarten-Grade 5) 


K-5 Cross-age mentoring Southern United > Mentoring Green, Mitchell, https://eric.ed.gov/?id=EJ 
and artist modeling States > Modeling & Taylor (2011) 932846 
PK, K Visual literacy New York City, Visual literacy interviews Lopatovska, https://www.researchgate. 
education NY/New Jersey Hatoum, net/publication/3086554 
Waterstraut, 37 Not just_a_ pretty pict 
Novak, &Sheer ure visual literacy educati 
(2016) on_ through art for young 
children 


Middle and Mixed Grade Levels (up to Grade 8) 


8 Improving student Midwest > Professional art Benton (2000) https://files.eric.ed.gov/full 
attitudes and gallery text/ED444921.pdf 
achievement in art > Lessons 

> Student portfolios 

K-8 Visual Thinking Byron, MN > School art teachers Housen (2001) _https://files.eric.ed.gov/full 
Strategies (VTS) > Curriculum designed text/ED457186.pdf 
curriculum to develop critical 

thinking 


> In-school discussions 
about art along with 
museum visits 


6 Peace Park Project Western United > Professional artists Krensky (2001) —http://www.artsedsearch.o 
States > Youth arts rg/study/going-on-beyond- 
organization zebra-a-middle-school-and- 


community-based-arts- 
organization-collaborate- 


for-change 
8 Online-learning Georgia > Online learning Lovin & Lambeth https://eric.ed.gov/?id=EJ 
component created by incorporating peer (2014) 1097404 
the teacher-researcher discussion groups 
> Digital resources 
6 The sequential method Washington > Art specialist Snow & https://eric.ed.gov/?id=EJ 
of teaching > Two 55-minute McLaughlin 718125 
perspective drawing lessons on various (2005) 
concepts of 
perspective drawing. 
6-8 Choice-Based Art Philadelphia, PA > Student-centered Varian (2016) https://eric.ed.gov/?id=ED 
Education (CBAE) inclusive curriculum 567778 
curriculum that supports the 
needs of diverse 
learners 


> 40-minute class five 
days a week for 36 
days 


$s 
American Institutes for Research 30 


Review of Evidence: Arts Education Through the Lens of ESSA 


Grade | Name of Intervention | Geographic Intervention Abstract Link 
Level Location Components 


Supplemental 
synchronous lessons 
to Florida Virtual 
School’s M/J 
Orientation to Art 2D 
course 


Florida 


> 
> 


High School and Mixed Grade Levels (up to Grade 12) 


10-12 


9-12 


B12 


4141 


8-12 


11& 
12 


=12 


9-12 


8-12 


The Artistic Impetus 
Model 


Online art community: 
New Media & Social 
Networking in Art 


Education 


Online collaborative 
music creation 
technologies 


Art-making and racial 
climate 


Activities based on 
Olivia Gude’s 
postmodern principles 


Studio art class 
designed to enhance 
creativity and thinking 
skills 


Introductory high 
school visual arts 


infused with labor 
studies content 


Use of computer 
hypertext (through a 
computer program 
called Storyspace) as a 
base for art education 


Concept-based inquiry 
in an arts classroom 


Jersey City, NJ 


Not specified 


Miami, FL 


Philadelphia, PA 


Freeland, PA 


Midwest 


Bronx, NY 


Not specified 


Not specified 


> 
> 


Online arts educator 


Web conferencing tool 
Blackboard 
Collaborate 


Lessons 


Sensory-based 
exercises 


> Reflective dialogues 


School art instructors 


> Custom designed 


social networking site 


Curriculum designed 
by the teacher- 
researcher 


Computer software 


> Online collaborative 


> 


tools 


Classroom art teacher 


> 8-week art classes 
> Art projects focused 


on identity and racial 
identity 


> Classroom projects 


Visual art journal kit 


> Classroom art teacher 
> Lessons emphasizing 


problem solving and 
creativity thinking 
skills 


> School art teacher 


University teacher 
educators 
Curriculum that 
includes artworks 
pertinent to labor 
studies 


The Storyspace software 
program for creating, 
editing, and reading 
hypertext fiction. 


> 
> 
> 


Classroom art teacher 
Critiques 

Two- or 3-day 
workshops 


Werner (2011) 


Amorino (2009) 


Castro (2012) 


Cremata & 
Powell (2017) 


Davis (2016) 


Ferry (2016) 


Harper (2003) 


Sosin, Bekkala, 
& Pepper- 
Sanello (2010) 


Taylor (2000) 


Walker (2014) 


Note. The studies in the table are listed alphabetically by first author within each grade level. 


a 
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https://eric.ed.gov/?id=ED 
549416 


https://eric.ed.gov/?id=EJ 
867993 


https://eric.ed.gov/?id=EJ 
976281 


https://eric.ed.gov/?id=EJ 
1138698 


httos://files.eric.ed.gov/full 
text/ED567779.pdf 


https://eric.ed.gov/?id=ED 
Bomii@ 


https://eric.ed.gov/?id=ED 
479394 


https://eric.ed.gov/?id=EJ 
1093573 


https://www.tandfonline.co 
m/doi/abs/10.1080/0039 
3541.2000.11651688 


https://eric.ed.gov/?id=EJ 
1040853 
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Evidence in Studies of Dance Interventions 


Four studies of dance interventions were classified as providing evidence aligned with ESSA 
tiers. Two studies produced evidence aligned with Tier III (Table 3.3), and the other two studies 
provided Tier IV evidence (Table 3.4). 


One study (Lobo & Winsler, 2006) that provided Tier Ill evidence assessed the effects of a 
creative dance and movement program on the social competence of preschool children in the 
Head Start setting. The study reported statistically positive findings from a well-designed and 
well-implemented randomized controlled trial (RCT). The study therefore meets a key design 
requirement for Tier |. However, because the study was based on a small sample (40 students) 
and took place at one site, it does not meet the large-sample and multisite requirements for Tier 
| evidence. The study was classified as providing Tier Ill evidence instead. 


Minton (2003) examined the effects of dance education on students’ creative thinking by 
comparing the creative thinking ability of students taking dance classes with the ability of 
students not taking dance classes in six high schools. Creative thinking ability was measured 
using the Torrance Test of Creative Thinking (TTCT). The dance classes included a mixture of 
dance forms at the beginning and advanced levels, and the time students spent on dancing each 
week varied across classes. The study found statistically significant positive effects on two of the 
TTCT subscales (originality and abstractness of titles). 


Two studies of dance interventions failed to meet the design requirements for Tiers I-III, but they 
included a research or theory-based rationale for the interventions. The studies therefore were 
classified as providing evidence at Tier IV. Harding and Haven (2009) examined the effect of 
peer coaching on transfer of skills and content into performance in dance classes in two public 
high schools. Leijen, Admiraal, Wildschut, and Simons (2008) investigated students’ learning 
experience in an international+2 distance dance education program delivered in an e-learning 
format using a virtual learning environment platform. 


12 The study was considered eligible because a quarter of the students in the sample were from the United States. 
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Table 3.3. Evidence Review Results: Studies of Dance Interventions With Tiers II and III Evidence 


Outcomes for 
Which Positive 


Name of Geographic Intervention Effects Were Evidence Characteristics of 
Intervention Location Components Found Tier Study Samples Citation Abstract Link 
PK Creative Mid-Atlantic > Small-group Social-emotional Tier Ill > AO preschool children Lobo & Winsler (2006) http://www.artsedsear 
dance/movement region creative learning: Social > 67% Hispanic, ch.org/study/the- 
program dance competence 16% African American, effects-of-a-creative- 
instruction 5% Asian, 7% Arab dance-and-movement- 
> Dance 5% Caucasian/Other PiOeta tion Bie SOc 
teacher competence-of-head- 
start-preschoolers/ 
9-12 High school dance Not specified » Dance Process abilities: Tier III 286 students across six Minton (2003) https://eric.ed.gov/?id 
classes teachers Creative thinking high schools =EJ824162 
> Amixture of 
dance 
forms at 
various 
levels 


Table 3.4. Evidence Review Results: Studies of Dance Interventions With Tier IV Evidence 


Name of Geographic Intervention 
Intervention Location Components ___ Citation Abstract Link 
11&12 Peer coaching in Suburban Group or Harding & Haven (2009) _ https://www.tandfonlin 
dance classes partner e.com/doi/abs/10.108 
coaching 0/15290824.2009.103 
87385 
Not E-Learning and International >» Dance Leijen, Admiraal, https://www.tandfonlin 
stated dance (use of including 14 teachers Wildschut, & Simons e.com/doi/abs/10.108 
Claroline Learning iS. students > Dance (2008) 0/146478908020879 
Environment) a program 51 
with four e- 
courses per 
student 


a 
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Evidence in Studies of Drama Interventions 


Our review found eight studies of drama interventions that provided evidence at Tiers III and IV. 
Two studies produced evidence aligned with Tier Ill (Table 3.5), and six studies provided Tier IV 
evidence (Table 3.6). 


Greene, Erickson, Watson, and Beck (2017) and Kisiel et al. (2006) both provided Tier III 
evidence. Greene et al. (2017) reported the results of five random assignment experiments in 
which classes were assigned by lottery to attend a live theater performance or to watch a movie 
version of the same story. The intervention examined in the study did not include anything 
beyond the opportunity to see live theater. The study found that students in the classes assigned 
to see live theater scored significantly higher on measures of tolerance and content knowledge 
(e.g., stronger command of the plot and vocabulary of those plays) than students in classes 
assigned to watch a movie. Kisiel et al. (2006) evaluated the impact of Urban Improv, a theater- 
based youth violence prevention program developed for inner-city youth, on a range of social- 
emotional outcomes (e.g., aggressive behaviors, prosocial behaviors, cooperation, and self- 
control) using a quasi-experimental design. The study found statistically significant positive 
effects of Urban Improv on measures of students’ cooperation, self-control, hyperactivity, and 
internalizing behavior, based on ratings provided by teachers. 


Six studies of drama interventions provided rationales needed for Tier IV evidence (Table 3.6). 
Four studies provided rationales linking specific drama interventions to various social-emotional 
learning outcomes such as self-concept, problem behavior, social skills, self-efficacy, and anger 
inhibition (Bernstein, Ablow, Maloney, & Nigg, 2014; Freeman, Sullivan, & Fulton, 2003; 
Goldstein & Winner, 2012; Holloway & LeCompte, 2001). One report provided a rationale linking 
a drama intervention (videotaping as a means of self-monitoring) with students’ theater 
performance (Lan & Morgan, 2003). The other report provided a rationale linking a drama 
intervention (improvisational learning activities in drama classes) with students’ critical thinking 
skills (Chou & Shih, 2010). 
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Table 3.5. Evidence Review Results: Studies of Drama Interventions With Tier Ill Evidence 


Intervention 
Components 


Geographic 
Location 


Name of 
Intervention 


Outcomes for 
Which Positive 
Effects Were 
Found 


Evidence 
Tier 


Characteristics of Study Samples 


Abstract Link 


4-12 Live theater Arkansas 


performance 


Field trip to see live 
theater performance 


Interactive 
program design 
using structured 
theater 
improvisation 


4 Urban Improv Boston,MA > 


(Ul) 


> Ul intervention 
staff 


> 9-week 
developmentally 
appropriate 
curriculum 


a 


American Institutes for Research 


> Art learning: 
Content 
knowledge 


> Social- 
emotional 
learning: 
Tolerance 


> Social- 
emotional 
learning: Self- 
control, 
cooperation, 
hyperactivity, 
internalizing 
behavior 


Tier Ill 


Tier Ill 


94 classes, approximately 1,500 
students 


> 140 students from eight fourth- 
grade classrooms 


> 44.5% African American, 
27.7% Hispanic, 13.1% biracial, 
5.8% Asian, 5.8% Other, 
2.2% Caucasian, 0.7% Native 
American 


Greene, Erickson, 
Watson, & Beck 
(2017) 


Kisiel, Blaustein, 
Spinazzola, 
Schmidt, Zucker, & 
van der Kolk 
(2006) 


http://journals.sa 


epub.com/doi/abs 


/10.3102/001318 
9X18761034 


https://www.tandf 
online.com/doi/ab 


s/10.1300/J202v 
O5n02 03 
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Grade Level 


Name of Intervention 


Geographic Location 


Elementary Grades (Prekindergarten-Grade 5) 


3&4 Creative drama 


Middle Grades (Grades 6-8) 


6 Dramatic improvisational 
activity 
6-8 Arts Focus 


High School Grades (Grades 9-12) 


9&10 PlayWrite 


High school Acting training 
freshmen 

(Grade 9) 

High school Self-monitoring 


Rural 


Cambridge, MA 


Not specified 


Midsize Urban 
Alternative high schools 


Suburban and large 
urban high schools for 
the arts 


Urban 


Intervention Components 


> Creative drama instruction 
> Planning by drama and music 


teachers 


Lessons 

Class discussion 

Intensive art instruction (90- 
minute daily) 

Innovative teaching 


> Workshops 
> Playwriting with coaches 
> Live performance 


> 


> 
> 
> 


High-intensity acting training 
(9 hours per week) 


Productions 

Play development 
Videotaping 
Self-monitoring 


Note. The studies in the table are listed alphabetically by first author within each grade level. 


$s 
American Institutes for Research 


Freeman, Sullivan, & Fulton 
(2003) 


Chou & Shih (2010) 


Holloway & LeCompte (2001) 


Bernstein, Ablow, Maloney, 
& Nigg (2014) 


Goldstein & Winner (2012) 


Lan & Morgan (2003) 


Abstract Link 


https://www.tandfonline.com/doi/ 
abs/10.1080/00220670309598 
801 


https://eric.ed.gov/?id=EJ 1093600 


https://eric.ed.gov/?id=EJ631870 


https://www.tandfonline.com/doi/ 
abs/10.1080/15401383.2014.9 
02342 


httos://www.tandfonline.com/doi 
abs/10.1080/15248372.2011.5 
73514 


https://www.tandfonline.com/doi 
abs/10.1080/00220970309602 
070?journalCode=vjxe20 
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Evidence in Studies of Music Interventions 


The majority of studies that were relevant to this evidence review involve music interventions. 
Our review found 50 studies of music interventions that provided evidence at Tiers II-IV. One 
study produced evidence aligned with Tier Il, eight studies produced evidence aligned with Tiers 
lll (Table 3.7), and 41 studies provided Tier IV evidence (Table 3.8). 


The study that provided Tier Il evidence (Helmrich, 2010) compared the algebra achievement of 
students receiving formal instrumental or choral music instruction during middle school (the 
instrumental group or the choral group) with those who received neither of those modes of musical 
instruction (the comparison group) in six school districts in Maryland. The study found that students 
in the choral group outperformed the comparison group, and the difference was statistically 
significant. This finding was considered to provide Tier Il evidence because the two groups were 
shown to be equivalent at baseline.13 


Five studies provided Tier Ill evidence about the impact of music interventions on students’ 
academic achievement outcomes (Gromko, 2005; Miksza, 2010; Piro & Ortiz, 2009; Sharpe, 
2013; Southgate & Roscigno, 2009).14 Three studies (Gromko, 2005; Piro & Ortiz, 2009; Sharpe, 
2013) used quasi-experimental designs and two (Miksza, 2010; Southgate & Roscigno, 2009) 
were correlational studies. Gromko (2005) found that kindergarten children who received 4 
months of music instruction showed significantly greater gains in their phoneme segmentation 
fluency than children who did not receive music instruction. Miksza (2010) examined the 
relationships between participating in high school music ensembles and students’ mathematics 
achievement, community ethic, and commitment to school. The study found that students 
participating in band, chorus, or orchestra had significantly higher math achievement scores and 
significantly higher levels of community ethic and commitment to school than students who did 
not participate in those activities. Piro and Ortiz (2009) found that students who participated in a 
scaffolded music instruction program had significantly higher vocabulary and verbal sequencing 
scores than students in the comparison group.15 Sharpe (2013) found that Grade 6 students 
who received music instruction through band, chorus, or orchestra during the school year scored 
significantly higher in mathematics achievement tests than students who did not receive music 
instruction. Using nationally representative samples, Southgate and Roscigno (2009) found that 
in-school music participation1s was positively associated with mathematics and reading 
achievement among children and adolescents. 


Among the other three studies that provided Tier Ill evidence, one study focused on art learning 
outcomes (Johnson, 2011), one study focused on process abilities (Rauscher & Zupan, 2000), 


13 The study also found statistically significant positive group differences in algebra achievement based on other 
comparisons (e.g., comparison between the instrumental group and the neither-instruction group, and subgroup 
comparisons). However, those findings did not meet Tier II criteria because the groups compared were not shown to be 
equivalent at baseline. Those findings were considered providing Tier IIl evidence because the author included 
statistical controls (including the pretest scores) in the analyses. 

14 One study (Miksza, 2010) also provided Tier III evidence on social-emotional learning outcomes. 

15 This intervention was also described in another study report (Piro, 2009), which was rated as providing no tier- 
aligned evidence. 

16 The study also examined music involvement outside of school and parental involvement in the form of concert 
attendance, which is out of the scope of this review. Our review focused on only in-school music involvement. 
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and the other study focused on social-emotional learning outcomes (Ritblatt, Longstreth, 
Hokoda, Cannon, & Weston, 2013). Johnson (2011) examined the effects of including critical 
thinking opportunities in music instruction (compared with activity-based music instruction that 
did not include opportunities for critical thinking) on music listening skills of Grade 5 students. 
Statistically significant differences were found between the treatment and comparison groups on 
the music listening and critical thinking measures. Ritblatt et al. (2013) assessed the effects of 
the Circle of Education, a music-based school-readiness program, on preschool children’s social 
skills and school-readiness skills. The study found that the improvement in social skills (total 
score) and specifically on the social cooperation, social interaction, and social independence 
scales were significantly higher for the music group than for the comparison group. Rauscher 
and Zupan (2000) reported the results of a field experiment in which kindergarteners were 
assigned to receive music instruction featuring the keyboard or to receive no music instruction. 
The study found that the keyboard group scored significantly higher than the no-music group on 
spatial-temporal tasks after 4 months of lessons, and that the difference was significantly 
greater after 8 months of lessons. 


Forty-one studies of music interventions did not meet the criteria for Tiers I-III evidence but did 
provide rationales needed for Tier IV evidence (Table 3.8). These studies provided rationales 
linking music education interventions to multiple outcome domains examined in this review. For 
example, 21 studies provided rationales linking music education interventions to art learning 
outcomes, 11 studies linked music education to social-emotional learning outcomes, eight studies 
linked music education to student achievement outcomes, eight studies linked music education to 
process abilities, eight studies linked music education to attitudes, and four studies linked music 
education to other outcomes within the scope of this review (e.g., later life involvement in music, 
outlook for life, and instructional strategies). 
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Table 3.7. Evidence Review Results: Studies of Music Interventions With Tiers II and Ill Evidence 


Outcomes for 
Which Positive Evidence 
Effects Were Found Tier 


Characteristics of 
Study Samples 


Intervention 
Components 


Grade Name of 
Level Intervention 


Geographic 
Location 


Citation Abstract Link 


Elementary Grades (Prekindergarten-Grade 5) 


K 


PK 


$s 
American Institutes for Research 


Music 


instruction States 


Critical Thinking Southwest 
United States 


Instruction (CTI) 


Keyboard 
instruction 
based on the 
Music and the 
Brain project 
curriculum 


Classroom 
keyboard 
instruction 


States 


Circle of 
Education 
music-based 
school- 
readiness 
program 


New York City 


Midwest United 


Southwest 
United States 


Midwest United >» Classroom 


teachers 
> University music 
method students 
> 4 months of music 
instruction 


Music instruction 

using open-ended, 

higher order cognitive 

questions and 

improvisation 

exercises 

> Music teacher 

> The Music and the 
Brain project 
curriculum 

> Aspecially 
equipped music 
laboratory 


> Music specialist 

> 20-min lessons 
twice per week in 
groups of 
approximately 
10 children 


>» 20 songs created 
based on the 
Desired Results 
Developmental 
Profile focusing on 
social-emotional 
skills 


> Child care 
instructors 


Academic 
achievement: 
Phoneme - 
segmentation 
fluency 


Art learning: Music 
listening skills 


Academic 
achievement: 
Vocabulary, verbal 
sequencing 


Process abilities: 
Spatial-temporal 
reasoning 


Social-emotional 
learning: Social 
cooperation, social 
interaction, social 
independence 


Tier Ill 


Tier Ill 


Tier Ill 


Tier Ill 


Tier Ill 


103 kindergarteners Gromko (2005) http://journals.sagepub.co 
at two elementary m/doi/abs/10.1177/002 
schools 242940505300302 


81 fifth-grade Johnson (2011) http://journals.sagepub.co 


students m/doi/abs/10.1177/002 
2429411415860 
> 103 second- Piro & Ortiz http://journals.sagepub.co 
grade students (2009) m/doi/10.1177/0305735 
608097248 


> 54% non-White 


> 53% receive free 
lunch 


> 62 middle-income Rauscher & Zupan http://psycnet.apa.org/rec 


kindergarten (2000) ord/2001-03185-003 
children 

> 36 boys and 26 
girls 


> Mixed ethnicity 


102 preschool 
students 


Ritblatt, Longstreth https://www.tandfonline.c 

Hokoda, Cannon, & om/doi/abs/10.1080/02 

Weston (2013) 568543.2013.796333?sc 
roll=top&needAccess=true 
&journalCode=ujrc20 
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Outcomes for 


Grade Name of Geographic Intervention Which Positive Evidence Characteristics of 
Level Intervention Location Components Effects Were Found Tier Study Samples Citation Abstract Link 


Middle Grades (Grade 6-8) 


6-8 Middle school Maryland Instrumental or choral Academic Tier Il 6,026 students Helmrich (2010) _http://journals.sagepub.co 
music programs instruction all 3 years achievement: across six school m/doi/10.1177/0743558 
during middle school Algebra districts 410366594 
achievement 
6 Music Atlanta, GA Music instruction Academic Tier Ill 109 sixth-grade Sharpe (2013) https://eric.ed.gov/?id=ED5 
Instruction and through band, chorus, achievement: Math students 63908 
Academic or orchestra achievement 
Achievement in 
Mathematics 


High School and Mixed Grade Levels (up to Grade 12) 


10 School music National High school music > Academic Tier Ill 12,160 students Miksza (2010) http://www. jstor.org/stable/ 
ensemble ensembles achievement: across 603 high 41110431 
participation Math schools 


achievement 


> Social-emotional 
learning: 
Community ethic, 
commitment to 


school 
K-14, Music National School music Academic Tier Ill > 4,376 students in Southgate & http://psycnet.apa.org/rec 
8-12 involvement programs achievement: the ECLS-K Roscigno (2009)  ord/2009-00725-001 
and academic Reading and math sample 
achievement achievement > 7,781 students in 
the NELS:88 
sample 


Note. The studies in the table are listed alphabetically by first author within each grade level. ECLS-K refers to the Early Childhood Longitudinal Study, Kindergarten Class data; 
NELS:88 refers to the National Education Longitudinal Study of 1988 data. Both data sets were collected and maintained by the National Center for Education Statistics. 
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Table 3.8. Evidence Review Results: Studies of Music Interventions With Tier IV Evidence 


Grade Level Name of Intervention Geographic Location Intervention Components Abstract Link 


Elementary Grades (Prekindergarten-Grade 5) 


5 Multicultural 4 Urban schools > Music concept with formal Abril (2006) http://journals.sagepub.com/doi/pdf/10. 
instructional approaches elements of music instruction 1177/0255761406063103 


> Sociocultural music instruction 


PreK Music therapy Not reported Using instrument play, body Barnes (2010) https://eric.ed.gov/?id=ED527813 
movement, listening, and 
vocalizations in music therapy 


PreK Pre-K music instruction | Southeast United States > Curriculum based on the Bowen (2010) https://eric.ed.gov/?id=ED519022 
National Music Standards for 
prekindergarten 


> Music instruction 


3 Mathematics-related Southeast GA > Music instruction Brock & Lambeth https://eric.ed.gov/?id=ED565237 
music instruction > Classroom music teacher (2013) 

2-4 Elementary music Midwest > Various activities: listening Coulson & Burke http://journals.sagepub.com/doi/abs/10 
activities and exercises homework exercise, in class (2013) .1177/0255761413495760 
for creativity activity 


> Teacher played music and 
demonstrated skills 


K-5 Use of a recorded aural Not specified > School music teacher Frewen (2010) https://eric.ed.gov/?id=EJ877133 
model > Recorded aural model 
K&1 Pedagogical approaches Suburban, CT > Traditional folk songs Gault (2002) https://www.jstor.org/stable/40319126 
and present/absence of > Four treatment conditions ?seq=1#page scan tab contents 
text in music instruction varying in the way songs were 
taught 


> Varying holistic teaching and 
text conditions (i.e., presence 
or absence of text) 


5 Self-monitoring: Rural, Midwest Use of student self-monitoring Hayes (2002) https://files.eric.ed.gov/fulltext/ED4715 
teacher/researcher checklist and other supports 85.pdf 
designed action toward developing and using 


poeearcn higher order thinking skills 
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Grade Level 


1-3 


PreK 


4&5 


PreK & K 


PreK 


$s SS 
American Institutes for Research 


Name of Intervention 


Intensive music program 


Musical ensembles in 
residence 


Harmony Project 


Music training in Head 
Start setting 


SingingCoach software 


Instruction and melody 
and improvisation 


Live Music 
Between the Lions TV 


Solfege syllables and 
relating tonal patterns 


Teacher feedback and 
modeling 


Early childhood music 
class 


Kindermusik 


Geographic Location 


New England 


Southwestern United 
States 


Los Angeles, CA 


Not specified 


Midwest to Northwest 
states 


Georgia 


Florida 


Northeast 


Pennsylvania 


Northeastern United 
States 


> MidAtlantic 


> Suburban or 
metropolitan area 


Intervention Components 


Increased time in general music 
classes (two to seven times more 
than control students) 


Enhancement to regular music 
program with ensemble 
interaction once a week 

> Community music program 
> Curriculum 

> Music instruction 

> Music training 

> Music activities 

> Classroom teachers 
Secondary data analysis of music 
classrooms participating in the 
Great American Singing 
Challenge 

> School music teacher 

> Instruction 


> Music groups 
> Television program 


Sight singing instruction with 


related and nonrelated (syllable) 
songs 


School music teacher 


Music class collaboration 


Music-based curriculum 
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Hogan, Cordes, 
Holochwost, Ryu, 
Diamond, & Winner 
(2018) 


Johnson & Davis 
(2016) 


Kraus, Hornickel, 
Strait, Slater, & 
Thompson (2014) 


Neville et al. (2008) 


Paney & Kay (2015) 


Parisi (2004) 


Register (2004) 


Reifinger (2012) 


Rutkowski & Miller 
(2003) 


St. John (2006) 


Winsler, Ducenne, & 
Koury (2011) 


Abstract Link 


https://doi.org/10.1016/j.ecresq.2017. 
12.004 


https://openmusiclibrary.org/article/172 
629/ 


https://www.ncbi.nim.nih.gov/pubmed/2 
5566109 


http://lup.lub.lu.se/record/3364201 


http://journals.sagepub.com/doi/full/10. 
1177/8755123314548047 


https://eric.ed.gov/?id=EJ8 45396 


https://academic.oup.com/jmt/article- 
abstract/41/1/2/1035029?redirectedFr 
om=fulltext 


https://www.jstor.org/stable/41348850 
?seq=1#page scan tab contents 


https://www.jstor.org/stable/40319169 
?seq=1#page scan tab contents 


http://journals.sagepub.com/doi/10.117 
7/0305735606061854 


https://www.tandfonline.com/doi/abs/1 
0.1080/10409280903585739 
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Grade Level 


5-8 


5&6 


$s 
American Institutes for Research 


Name of Intervention 


Geographic Location 
Middle and Mixed Grade Levels (up to Grade 8) 
6-8 


School-loaner instrument Urban 


program 


Self-evaluation 
instruction 


Peer Assisted Learning 
(PAL) structures 


Music education 
programs 


Peer Connectedness 
Study 


Teaching composition in 
middle school band 


Music education and 
mentoring 


Melodic error detection 


A curriculum to increase 
music achievement 
using technology, self- 
assessment, and 
positive feedback 


Metropolitan mid- 
Atlantic 


Urban/suburban 


South, East Coast, 
Midwest, West Coast 


Suburban, upper 
Midwest 


Not specified 


Midwest 


Not reported 


Urban 


Intervention Components 


Providing instruments to students Ester & Turner (2009) 
who could not afford to own their 


own 


Instruction in self-evaluation 


Symmetrical and asymmetrical 


(PAL) groups 


> Music programs 
> Music instruction 


School-based instrumental music 
instruction and band-related 


activities 


> Teaching composition along 


with performance and 
listening to wind and 
percussion players 


> Use of SCAMPER strategies in 


some activities 


General instruction alternates 
with performance-oriented and 


product-oriented 


> Listening to themselves while 


playing 
Listening to recordings 


vy 


Familiar melodies 


Music Shop software 
Portfolio development 


vveviyv 


Pentachord unit 


MIDI capable workstations 
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Hewitt (2011) 


Johnson (2017) 


Johnson & Memmott 
(2006) 


Rawlings & Stoddard 
(2017) 


Riley (2006) 


Shields (2001) 


Thornton (2008) 


Vega (2001) 


Abstract Link 


httos://eric.ed.gov/?id=EJ864262 


http://journals.sagepub.com/doi/10.117 
7/0022429410391541 


htto://journals.sagepub.com/doi/abs/10 
.1177/0022429417712486 


https://eric.ed.gov/?id=EJ773246 


http://journals.sagepub.com/doi/abs/10 
.1177/1321103X17703575 


http://journals.sagepub.com/doi/abs/10 
.1177/87551233060250010104 


http://journals.sagepub.com/doi/pdf/10. 
2307/3345712 


http://journals.sagepub.com/doi/pdf/10. 
dnMni/iSia SUZ SS0SsamGuk2 


httos://files.eric.ed.gov/fulltext/ED4560 
86.pdf 
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Grade Level Name of Intervention Geographic Location Intervention Components Abstract Link 
Middle school Little Kids Rock Dallas, TX > Free instruments Thomas (2016) https://www.arts.gov/sites/default/files/ 
> Teacher training Research-Art-Works-MississippiState.pdf 


> Curricular materials 
> Music instruction 


High School and Mixed Grade Levels (up to Grade 12) 


Adolescents Music mentorship Not identified Music with mentorship of Darrow, Novak, https://eric.ed.gov/?id=EJ912418 
program students with developmental Swedberg, Horton, & 
disabilities Rice (2009) 

High school Music class participation National School music offerings Elpus (2013) http://journals.sagepub.com/doi/abs/10 
.1177/0022429413485601?journalCod 
e=jrma 

9-12 Comprehensive Musician _ Illinois > School band directors Gustafson-Hinds http://irl.umsl.edu/dissertation/486 

(CMP) > Lesson plans (2010) 
> Unit study-technology 
approach 

9-12 Music history and theory Not reported Music history and theory lessons — Hendricks (2010) http://www.jstor.org/stable/2 7861483 

instruction 

9-12 Sight-singing instruction Central Texas > School music teacher Henry (2004) https://eric.ed.gov/?id=EJ 708660 

and specific pitch skills > Solfege drill 
7-9 Modeling, self- Southwest Varied combinations of modeling, Hewitt (2001) http://journals.sagepub.com/doi/10.230 
evaluation, and self- self-listening, and self-evaluation 7/3345614 
listening before and during 
instrumentalists’ music 
performance 
9-12 The Lakewood Project Midwest > Use of classical instruments Koops, Hankins, http://journals.sagepub.com/doi/full/10. 
and rock band instruments Scalise, & Schatt 1177/1321103X14537432 
> Focus on informal learning (2014) 
integrated with traditional 
teaching 
Mixed high Music composition Midwest > Composition instruction Menard (2015) http://journals.sagepub.com/doi/abs/10 
school instruction > Teacher orientation .1177/0022429415574310 
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Grade Level Name of Intervention Geographic Location Intervention Components Abstract Link 
6-12 Using recorded models > Rural, urban, and Recorded models of professional Morrison, http://journals.sagepub.com/doi/10.230 
in ensemble rehearsals suburban recordings used to prepare for Montemayor, & 1/3345434 
> Pacific Northwest selected pieces Wiltshire (2004) 
6-12 Mariachi Education Nevada, Georgia, > Curriculum Neel (2017) https://eric.ed.gov/?id=EJ1154274 
Program Tennessee, California S Taadner PD 


> Music instruction 


High school High school music Chicago, Illinois > Two years of high school Tierney, Krizman, http://www.frontiersin.org/journal/10.33 
classes music classes Skoe, Johnston, & 89/fpsyg.2013.00855/abstract 
Kraus (2013) 
Adolescents Weill Music Institute’s > Urban > Twelve sessions over 2 weeks Wolf & Holochwost http://wolfbrown.com/images/books_rep 
strength-based music > Northeast > Music focus (2014) orts/documents/ourvoicescount.pdf 


program in juvenile 
justice settings 


Note. The studies in the table are listed alphabetically by first author within each grade level. 
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Evidence in Studies of Interventions That Involve Multiple Types 
of Art 


Seven studies provided evidence aligned with ESSA tiers on interventions that involved more 
than one type of art. Two studies provided evidence at Tier Il, two studies provided evidence at 
Tier Ill, and three studies provided evidence at Tier IV. 


Both interventions supported by Tier Il evidence took place in New York City (Table 3.9). Chen, 
Lui, Andrade, Valle, and Mir (2017) examined the effect of professional development in the 
use of criteria-referenced formative arts assessment on students’ achievement in arts. The 
study was part of the Arts Achieve project, funded by a U.S. Department of Education Investing 
in Innovation Fund grant and an Arts in Education Model Development and Dissemination 
grant. The treatment group consisted of students taught by teachers in all four major art 
disciplines (i.e., music, visual arts, drama, and dance) who received the professional 
development. The control group consisted of students in classes instructed by teachers who 
did not receive the professional development. The study showed that professional 
development in the use of the criteria-referenced formative assessment in arts had a 
statistically significant positive effect on students’ arts achievement, as measured by 
discipline-specific, performance-based assessments. 


Horowitz (2016) examined the impact of the Everyday Arts for Special Education (EASE) 
program—also funded through an Investing in Innovation Fund development grant to New York 
City’s special education district (District 75)—on special education students’ academic 
achievement and social-emotional learning. The participating EASE students were in grades 2-5 
in 10 schools. EASE teachers participated in professional development and received in-school 
support on strategies across multiple arts disciplines (i.e., music, dance, visual arts, and theater). 
The study found statistically significant effects on social-emotional learning outcomes and on 
reading achievement. However, only the findings on social-emotional outcomes met the criteria 
for Tier Il evidence; the findings on reading achievement were based on a sample that is smaller 
(N = 157 students) than the threshold for Tier Il evidence and were therefore classified as 
providing Tier Ill evidence. 


Like Chen et al.’s (2017) study, Mastrorilli, Harnett, and Zhu (2014) also examined the impact of 
professional development in the use of criteria-referenced formative assessment in arts as part 
of the Arts Achievement project, but with a slightly different sample. The study provided Tier III 
evidence on students’ arts achievement. Neddeau (2013) provided Tier Ill evidence on the 
relationship between teachers’ reported curriculum emphasis in arts (music, art, dance or 
creative movement, and theater or creative dramatics) and students’ reading achievement. 
Teachers’ curriculum emphasis in arts was measured by instructional minutes per week. The 
study found a statistically significant positive relationship between teachers’ curriculum 
emphasis in arts and students’ reading achievement in Grade 5. 


Three studies of interventions involving multiple types of art did not meet the criteria for 
evidence at Tiers I-III but did provide rationales needed for Tier IV evidence (Table 3.10). Abedin 
(2010) investigated the learning experience of adolescents with learning disabilities in their 
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music and drama classes during an 8-week curriculum unit in a public charter school with a 
quality arts program. Lovett (2014) examined the relationship between fine arts courses (dance, 
music, theatre arts, and visual arts) offered in a school and the graduation rate of the school 
among Georgia’s public high schools. Rapp-Paglicci, Stewart, and Rowe (2012) evaluated the 
impact of the Prodigy Cultural Arts Program, an early prevention program for adjudicated youth, 
on students’ mental health, academic achievement, and family functioning. 
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Table 3.9. Evidence Review Results: Studies of Interventions Involving Multiple Art Types With Tiers Il and III Evidence 


Name of 
Intervention 


Intervention 
Components 


Outcomes for Which 
Positive Effects 
Were Found Tier 


Evidence Characteristics of Study 


Samples 


Citation 


Abstract Link 


K-12 PD on criteria- 
referenced 
formative 
assessment in 


the arts as part of 


the Arts Achieve 
project 


2-5 Everyday Arts for 


Special 


Education (EASE) 


K-12 PD on criteria- 
referenced 
formative 
assessment in 


the arts as part of 


the Arts Achieve 
project 


1,3,5 Curricular 
emphases and 
academic 
achievement 


Art teachers 
Teacher PD 


Teacher 
PLCs 


Teacher PD 
workshops 


On-site PD 
Collaborative 


classroom 
modeling 

Classroom 
instruction 


Teaching 
artists 


> Teacher PD 
> Teacher 


PLCs 


Classroom 
teachers 
Curricular 
emphasis in 
arts 


Art learning: Arts Tier Il 
assessment scores 


> Academic Tier Il 
achievement: 
Reading and 
math 
achievement 


> Social-emotional 
learning: 
Functional skills 


Art learning: Arts Tier III 
assessment scores 
Academic Tier Ill 
achievement: 

Reading 


achievement 


> 5,640 students across 75 
schools 


> 72% minority 


>» Special education students 


> Academic achievement 
analysis included 157 
Grade 4 students within 
10 schools 

> Analysis of social- 
emotional behavior 
included 759 students 
within 10 schools 


4,066 students 
85% minority 
11% English learners 


13% special education 
students 


> 85% receiving free or 
reduced-price lunch 


Students in Grade 5 ECLS-K 
sample 


vvvyv 


Chen, Lui, 
Andrade, Valle, & 
Mir (2017) 


Horowitz (2016) 


Mastrorilli, 
Harnett, & Zhu 
(2014) 


Neddeau (2013) 


Note. The studies in the table are listed alphabetically by first author. PD = professional development. PLC = professional learning community. 
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https://eric.ed.gov/?i 
d=EJ1152545 


https://eric.ed.gov/?i 
d=ED573547 


http://www.artsedse 
arch.org/study/h-the- 
arts-101-1-24/ 


https://repository.usf 
ca.edu/cgi/viewconte 


nt.cgi?article=1090& 
context=diss 
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Table 3.10. Evidence Review Results: Studies of Interventions Involving Multiple Art Types With Tier IV Evidence 


Grade Level Name of Intervention Geographic Location Intervention Components Citation Abstract Link 
7&8 Art-based education for >» Urban Art, music, and drama Abedin (2010) https://drum.lib.umd.edu/handle/190 
adolescents with > Public charter classes supporting 3/10505 
learning disabilities school curriculum unit on Middle 
Eastern and African arts 
9-12 Fine arts courses Georgia Dance, music, theatre Lovett (2014) https://eric.ed.gov/?id=ED557020 
arts, and visual arts 
courses 
Adolescents, Prodigy Cultural Arts Urban > Classes in performing Rapp-Paglicci, Stewart, & https://www.tandfonline.com/doi/abs/ 
ages 10-18 based on Positive Youth arts taught by master Rowe (2012) 10.1080/15433714.2011.581532 
Development Model artists from the 
community 
> Three hours per week 
for 8 weeks 
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Summary 


This chapter presents findings for the first research question that motivated this review: Are 
there research studies on arts education interventions that meet the criteria for evidence as 
specified in ESSA? These findings are summarized by type of art. 


For visual arts interventions: 


> One study provided Tier Ill evidence suggesting that the students who participated in two 
inner-city programs that featured rich, sustained visual arts made significant greater gains 
than comparison students in social-emotional learning outcomes and process abilities. 


> Eighteen studies provided Tier IV evidence by including a research- or theory-based rationale 
for how visual arts interventions should impact students’ art learning, social-emotional 
learning, process abilities, and attitudes toward arts. 


For dance interventions: 


> Two studies produced Tier Ill evidence on the positive effects of dance interventions on the 
creative thinking abilities of high school students and on the social competence of preschool 
children. 


> Two other studies provided Tier IV evidence supporting dance interventions by presenting 
research- or theory-based rationales for such interventions. 


For drama interventions: 


> Two studies provided Tier Ill evidence supporting drama interventions. One study found that 
students who were randomly assigned to see live theater scored significantly higher on 
measures of tolerance and art content knowledge than students in the control group. The 
other study provided evidence on the positive effects of Urban Improv, a theater-based youth 
violence prevention developed for inner-city youth, on a range of social-emotional outcomes. 


> Six studies provided Tier IV evidence supporting drama interventions by presenting research- 
or theory-based rationales for such interventions. 


For music interventions: 


> One study provided Tier Il evidence on the effect of formal instrumental or choral music 
instruction during middle school on students’ algebra achievement. 


> Eight studies provided Tier Ill evidence. Five studies examined the impact of music 
interventions on students’ academic achievement outcomes, one study focused on art 
learning outcomes, one study focused on social emotional learning outcomes, and the other 
study focused on process abilities. 


> Forty-one studies provided Tier IV evidence supporting music interventions by presenting 
research- or theory-based rationales linking music interventions to all outcome domains 
examined in this review. 
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For interventions that involved multiple types of art: 


> Two studies provided Tier Il evidence supporting the effectiveness of two such interventions 
implemented in New York City. One study examined the effectiveness of professional 
development in the use of criteria-referenced formative assessment in arts as part of the Arts 
Achieve project on students’ achievement in arts. The other study examined the impact of 
the Everyday Arts for Special Education program on special education students’ academic 
achievement and social-emotional learning. 


> Two studies provided Tier Ill evidence. One study assessed the effect of professional 
development in the use of criteria-referenced formative assessment in arts on students’ 
achievement in arts. The other study examined the relationships between teachers’ reported 
curriculum emphasis in arts (music, art, dance or creative movement, and theater or creative 
dramatics) and students’ reading achievement. 


> Three studies provided Tier lV evidence supporting arts education interventions that involved 
multiple types of art. That is, these studies include a research- or theory-based rationale for 
how such interventions should impact student outcomes. 
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Chapter 4. How Large Are the Effects of Arts 
Education Interventions on Student Outcomes? 


Chapter Highlights 


> The average effect found in well-designed and well-implemented studies (i.e., those 
having research designs capable of producing evidence at Tiers I-Ill) was statistically 
significant and moderate in magnitude. 


> The effects of arts education interventions varied by art type and outcome domain. 


While Chapter 3 focuses on the levels of evidence supporting arts education interventions, this 
chapter focuses on the size of the effects that educators might expect if they adopt an arts 
education intervention. Specifically, this chapter addresses the second research question for this 
evidence review: 


Research question 2: How large are the effects of arts education interventions on 
student outcomes? 


Although the classification of interventions based on ESSA evidence tiers is useful for policymakers 
to ensure that educational agencies implement interventions that are supported by research, this 
approach has several conceptual limitations. First, the classification is biased toward statistically 
significant findings. As a hypothetical example, a study can make 100 statistical comparisons 
between an arts education group and a comparison group, and 99 of the comparisons could show 
no statistically significant differences. However, the one statistically significant effect would be 
sufficient for the intervention to be classified as providing evidence at Tiers I-III, provided that 
other studies on the same intervention did not show a statistically significant negative effect. 


Second, the classification of interventions based on the ESSA evidence tiers is biased toward large 
studies and multisite studies. In one way, this bias makes sense in that larger studies can generate 
estimates of impact that are more accurate than smaller studies. However, the numbers of sites 
and individuals involved in large-scale studies may stretch the abilities of arts education 
intervention developers to monitor the quality of implementation and address deviations from the 
intended program model. In reviewing the research evidence, some allowance should be given to 
small studies that could have implemented arts education interventions with greater fidelity. 1” 


Third, for school and district administrators who are considering investing in an arts education 
intervention, the rating of the intervention in terms of the ESSA evidence tiers fails to provide 
them with an idea of the magnitude of improvement in student outcomes that can be expected 
when adopting the intervention. Such information requires an examination of all the effects 


17 In this review, we did not examine the fidelity of implementation of the interventions that were the focus of the 
studies reviewed. This is consistent with the convention used in the WWC review process. 


$s 
American Institutes for Research 52 


Review of Evidence: Arts Education Through the Lens of ESSA 


produced from well-designed and well-implemented studies of the intervention, regardless of 
whether they produced statistically significant findings. 


To address these limitations and better understand the expected effects of arts education 
interventions, we examined findings about the effects of arts education interventions from all 20 
studies that meet the design-related criteria+8 for Tiers I-III and contain effect estimates that could 
be converted to an effect size19 (or provided enough information for reviewers to calculate effect 
sizes and standard errors), even if their results did not meet the ESSA evidence standards. We 
examined, for example, findings that were not statistically significant, findings from studies with a 
sample size smaller than 350 students, and findings from studies conducted in a single site. 


For each of the 20 studies, we examined the magnitude of the differences in student outcomes 
between the arts education intervention group and the comparison group using meta-analysis 
procedures. We began by converting group differences in each study to the Hedges’ g statistic, which 
is an expression of the magnitude of the difference between two groups in terms of the standard 
deviation of the outcome measure. Conversion to the Hedges’ g allows us to compare the effects 
observed based on different outcome measures in different studies using a single, common metric. 
Box 4.1 provides a brief explanation of our meta-analysis procedures. See Appendix A for more 
details on the methods used. 


Box 4.1. Meta-Analysis Explained 


A meta-analysis is a set of quantitative procedures used to statistically combine the effects from multiple studies 
(Cooper, 2010). It can help researchers and policymakers better understand the magnitude of the effect that an 
intervention can have as well as the variability of effects found with different types of samples or settings. The 
meta-analysis that produced the findings presented in this chapter involved the following six steps: 


1. Recording details about each study. For each study that met the design criteria for ESSA’s Tiers I-III 
(regardless of the sample size or the number of sites of the study), the review team recorded the 
characteristics of the intervention, research design, study setting (e.g., urban vs. suburban vs. rural), sample 
characteristics (e.g., student demographic information), sample size, outcome measures, and statistics about 
the effect of the intervention on each relevant outcome. 


2. Converting effect statistics into a common effect size metric. The statistics reported in studies indicate the 
differences between students exposed to the intervention and students not exposed to the intervention on 
some outcome measure (or in some cases, the relationship between students’ exposure to arts education 
and an outcome). These statistics were converted into a common effect size metric—Hedges’ g, which 
represents a standardized mean difference in terms of the pooled standard deviation of the outcome 
measure. (The figure below illustrates an effect size of 0.50 in terms of Hedges’ g.) 


18 A study that meets the design-related criteria should be a well-designed and well-implemented RCT or quasi- 
experimental study (e.g., based on WWC standards) or a correlational study with statistical controls. The study may not 
have a large sample or a multisite sample as required for Tier | and Tier II evidence in the Non-Regulatory Guidance 
(U.S. Department of Education, 2016). 

19 The study needs to provide effect size estimates and standard errors or provide statistics that could be used to 
calculate effect sizes and standard errors. 
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Distribution of 
scores for arts 
education group 


Distribution of 
scores for 

comparison 
group 
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(-) Student achievement outcome (+) 


3. Determining the standard error for each effect size. Each effect size was estimated based on a particular 
sample, and there is some degree of uncertainty about whether the effect size estimate reflects the true 
effect for the population. The amount of uncertainty, measured by the standard error, reflects to a large 
extent the sample size of the study. All else being equal, the larger the sample, the more certainty we have 
about the effect estimate and the smaller the standard error. When the standard error for an effect size was 
not provided by the author(s), we calculated it using the formula provided by Hedges and Olkin (1985, p. 86). 


4. Averaging weighted effect sizes. To determine the average effect on a particular outcome across all 
relevant studies, we calculated the pooled effect size by estimating a standard random-effects meta-analytic 
model. The random-effects model tends to produce more conservative effect size estimates (i.e., estimates 
closer to 0) compared with a fixed-effects model. We estimated the models using a robust variance 
estimation method (Hedges, Tipton, & Johnson, 2010; Tanner-Smith, Tipton, & Polanin, 2016), which 
account for both the varying degrees of precision across studies and the nonindependence of effect sizes 
within studies. The robust variance estimation method upweights effect sizes that are estimated with greater 
precision and downweights estimates from studies that contribute multiple nonindependent effect size 
estimates. 


5. Calculating the confidence interval for the average effect. Even across multiple studies, there remains 
some uncertainty about the true magnitude of the effect for the population. Another statistic—the 95% 
confidence interval—gives the upper and lower bounds within which the true population effect is likely to lie. 
The narrower the interval, the more confident we are about the average effect size accurately reflecting the 
true effect for the population. The upper and lower bounds of the confidence interval were calculated as the 
average effect size +1.96 times the standard error of the effect size estimate. In this chapter, we use figures 
like the one below to indicate the magnitude of an average effect and the corresponding 95% confidence 
interval. 


Left-most point of diamond is Right-most point of 
the lower bound of confidence <--- —_ -----diamond is the upper bound 
interval of confidence interval 


Midpoint of diamond is 
estimate of average effect 


6. Identifying potential moderating factors. To determine whether effect sizes are related to art type or 
outcome domain, we calculated separate average effect sizes and confidence intervals by art type and 
outcome domain. We used standard meta-analytic procedures to determine whether the variation in effect 
sizes across art types and outcome domains exceeds what one would expect due to sampling error alone. 


We conducted the meta-analysis using the robumeta package in R, which is an open-source software 
environment for statistical computing and graphics. 


In the sections that follow, we first present the distribution of effect sizes and the average effect 
size across all the 20 studies included in our meta-analysis, by art type and overall. We then 
break down the findings according to the type of outcome examined (i.e., academic 
achievement, art learning, social-emotional learning, and process abilities). 
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Average Effect of Arts Education Interventions on Student 
Outcomes 


Overall, we found 77 effect size estimates from 20 studies that met the research design 
requirements for evidence at Tiers I-III. For studies reporting multiple effects, we first computed 
the average effect size within each study. The average within-study effect sizes from the pool of 
relevant studies ranged from 0.00 (Kisiel et al., 2006) to 1.40 (Piro & Ortiz, 2009; see Table 4.1). 
The overall average effect size across all studies was +0.38 (95% confidence interval = 
0.20-0.56), a moderate and statistically significant effect (see Table 4.1). 
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Table 4.1. Average Effects of Arts Education Interventions on Student Outcomes Within and Across Studies, by Art Type and Overall 


Art yperend Study Namie Effect Sizes 95% Confidence Interval 
(Hedges’) | ower Limit Upper 
Art (Visual art) 
Catterall & Peppler (2007) 0.82 0.66 0.98 = 
Dance —y— 
Lobo & Winsler (2006) 0.73 0.47 0.99 
Drama —{}— 
Freeman et al. (2003) 0.03 -0.21 0.27 —-—. 
Goldstein & Winner (2012) 0.44 -0.16 0.39 {4 
Kisiel et al. (2006) 0.00 -0.09 0.09 
Lan & Morgan (2003) 0.30 -0.38 0.96 
Music —i— 
Gromko (2005) 0.38 0.15 0.61 ia 
Helmrich (2010) 0.21 0.18 0.24 
Henry (2004) 0.44 -0.07 0.93 —i- 
Hogan et al. (2018) 0.01 -0.15 0.17 a. 
Johnson (2011) 0.98 0.52 1.44 [| 
Miksza (2010) 0.05 0.02 0.07 ———_— 
Piro & Ortiz (2009) 1.40 1.09 1.70 —- 
Rauscher & Zupan (2000) 0.83 0.62 1.05 —— 
Register (2004) 0.12 -0.22 0.46 + 
Ritblatt et al. (2013) 0.60 0.43 0.78 = 
Rutkowski & Miller (2003) 0.18 0.14 0.49 _——e 
Sharpe (2013) 0.42 -0.02 0.86 
Multiple types of art 5 
Chen et al. (2017) 0.26 0.15 0.37 i 
Horowitz (2016) 0.20 0.05 0.35 
Weighted average effect 0.38 0.20 0.56 O 
2 -1 0 1 2 
Favors comparison group Favors arts education group 


Note. The square for each study represents the average effect size across all relevant outcomes for the study, and the size of the square is proportional to the sample size. The 
horizontal line for each study shows the 95% confidence interval (i.e., amount of uncertainty) for the average effect. Squares with no lines indicate very precise estimates. 
Average effects with horizontal lines crossing the vertical line for zero are not statistically significant at the .05 level. 

a The effect size for each study represents a weighted average of all effect sizes reported in the study using the inverse of the variance of each effect estimate as the weight. 
The overall weighted average effect was estimated using a random-effects model with robust variance estimation. 

Source: Authors’ analysis of effects from studies meeting design requirements for ESSA Tiers I-III. 
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Table 4.2. Average Effects of Arts Education Interventions, by Art Type and Type of Student Outcomes 


Favors Favors Arts 
Average Comparison Intervention 
Number of Number of Effect Size Improvement Group Group 
Type of Art and Outcome Effects Studies (Hedge’s g) Index 95% Confidence Interval -1.0 -0.50 0 +0.50 +1.0 +1.5 +2.0 
Visual art 5 1 0.82 29 0.66 to 0.98 & 
Social-emotional learning 3 1 0.58 22 0.36 to 0.79 — 
Process abilities 2 1 1.13 37 0.89 to 1.38 > 
Dance 6 1 0.73 27 0.47 ~—s to 0.99 => 
Social-emotional learning 6 1 0.73 27 0.47 to 0.99 f —>S_- ——~—«t 
Drama 24 4 0.09 4 0.10 to 0.29 <—> 
Art learning al 1 0.30 12 0.35 to 0.95 —— 
Social-emotional learning 23 3 0.04 2 -0.09 to 0.18 © 
Music 38 12 0.45 17 0.17 to 0.72 | > 
Academic achievement 16 6 0.39 15 -0.08 to 0.85 | = ——“—‘=_ 
Art learning 6 3 0.57 21 -0.50 ‘to 1.63 
Social-emotional learning 10 3 0.20 8 -0.62 to 1.03 
Process abilities 6 al 0.83 30 0.62 to 1.05 —_ 
Multiple types of art 4 2 0.24 9 0.04 to 0.52 > 
Academic achievement 2 1 0.23 9 -0.01 to 0.47 <> 
Art learning 4 4 0.26 10 0.15 to 0.37 © 
Social-emotional learning 1 1 0.18 7 0.014 to 0.34 Sm 
All art types 77 20 0.38 15 0.20 to 0.56 | } 
Academic achievement 18 7 0.44 17 0.00 ‘to 0.88 _———— 
Art learning 8 5 0.43 17 0.00 ‘to 0.87 =— 
Social-emotional learning 43 9 0.23 9 -0.01 to 0.47 < 
Process abilities 8 2 0.93 32 -0.74 ~— to 2.60 


Note. The midpoint of each diamond indicates the point estimate for the average effect; the width of the diamond represents the 95% confidence interval. Diamonds that cross the 
vertical line for O are not statistically significant at the .05 level. When multiple studies provided evidence for an art type and outcome domain combination (i.e., each row in the table), a 
weighted average effect was calculated based on a random-effects model with robust variance estimation methods. Improvement index indicates the percentile point growth that would 
be expected for a student at the 50th percentile in the comparison group, had the student received the intervention. 

Source: Authors’ analysis of effects from studies that meet design requirements for ESSA Tiers I-III. 
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Average Effects for Visual Art Interventions 

Only one study (Catterall & Peppler, 2007) on visual art interventions meets the design 
requirements for ESSA Tiers I-lIll. The study produced five effect sizes eligible for inclusion in the 
meta-analysis: three on social-emotional learning outcomes (self-efficacy) and two on process 
abilities (originality). The average effect for each outcome domain is positive and statistically 
significant (Table 4.2). The average effect across all outcomes is also positive and statistically 
significant with a large magnitude (g = 0.82). Another way of looking at the magnitude of this 
effect is to consider what exposure to the intervention might mean to an average student in the 
comparison group. With an effect size of 0.82, such a student would likely move from the 50th 
percentile to the 79th percentile in the outcome (or an improvement index2° of 29 percentile 
points) had the student received the intervention. Across the 162 intervention effects on relevant 
outcome domains reported for the 70 interventions in mathematics, reading, and science 
reviewed by the WWC, the improvement indices ranged from negative 25 percentile points to 
positive 46 percentile points (see Figure 4.1 and further explanation in Appendix A). The 29- 
percentile-point increase would put the average effect of the visual art intervention at the 96th 
percentile among the 70 interventions reviewed by the WWC. 


Figure 4.1. Distribution of Improvement Indices for Intervention Effects on Student 
Achievement in Mathematics, Reading, and Science Based on WWC Intervention Reports 


NO 
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Source: Authors’ analyses and improvement indices found in WWC intervention reports. 


20 The WWC translates effect sizes into an improvement index, which reflects the percentile point gain that an average 
student (i.e., the student at the 50th percentile) in the comparison group would experience had he or she been 
exposed to the intervention. 
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Average Effect for Dance Interventions 

One study of a dance intervention (Lobo & Winsler, 2006) provided six effects eligible for 
inclusion in the meta-analysis, all in the social-emotional learning outcome domain. This study 
(Lobo & Winsler, 2006) is an RCT examining the effects of an 8-week instructional program in 
creative dance/movement on the social competence of low-income preschool children. The study 
found significantly greater positive improvements over time in children’s social competence and 
both internalizing and externalizing behavior problems for the experimental group compared with 
the control group (g = 0.73 for the average effect; improvement index = 27). 


Average Effect for Drama Interventions 

Four studies of drama interventions were included in the meta-analysis. One study examined 
students’ art learning outcomes (Lan & Morgan, 2003) and three studies examined students’ 
social-emotional outcomes (Freeman et al., 2003; Goldstein & Winner, 2012; Kisiel et al., 2006). 
These four studies produced a total of 24 effect sizes for the meta-analysis. The average effect 
across outcomes was not statistically significant (g = 0.09; improvement index = 4). Specifically, 
the only study that examined the effect of drama education on students’ art learning found a 
positive but nonsignificant effect (g = 0.30; Lan & Morgan, 2003). The three studies that examined 
the effects of drama education on students’ social-emotional learning outcomes had a small and 
nonsignificant average effect (g = 0.04; Freeman et al., 2003; Goldstein & Winner, 2012; Kisiel et 
al., 2006). 


Average Effect for Music Interventions 

A total of 38 effects from 12 studies of music interventions were included in the meta-analysis. 
When averaged across outcomes and interventions, the effect of music interventions on student 
outcomes is moderate and statistically significant (g = 0.45; improvement index = 17). One study 
(Rauscher & Zupan, 2000) examined the effects of a classroom music instruction featuring the 
keyboard on the process abilities (spatial-temporal reasoning) of kindergarten children. The 
study found the experimental group scored significantly higher than the comparison group on 
spatial-temporal tasks after 4 and 8 months of lessons (g = 0.83; improvement index = 30). 
Three studies each examined the effects of music education interventions on students’ art 
learning (Henry, 2004; Johnson, 2011; Rutkowski & Miller, 2003) and social-emotional learning 
outcomes (Hogan et al., 2018; Miksza, 2010; Ritblatt et al., 2013), and six studies (Gromko, 
2005; Helmrich, 2010; Miksza, 2010; Piro & Ortiz., 2009; Register, 2004; Sharpe, 2013) 
examined student achievement outcomes. When averaged across studies within each outcome 
domain, all three average effects were positive (ranging from 0.20 to 0.57), but none were 
statistically significant. 


Average Effect for Interventions Involving Multiple Types of Art 

Two studies included in the meta-analysis examined interventions that involved multiple types of 
art. Chen et al.’s (2017) study examined the effect of a professional development program 
focused on formative assessment practices in arts on students’ achievement in multiple art 
disciplines (music, visual arts, drama, and dance) in 75 schools across the New York city. The 
study found the intervention had a statistically significant positive effect (g = 0.26; improvement 
index = 10) on students’ arts achievement. Horowitz’s (2016) study assessed the impact of the 
Everyday Arts for Special Education program on special education students’ academic 
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achievement and social-emotional behavior in 10 schools in New York’s District 75. The study 
found a statistically significant average effect on social-emotional learning outcomes (g = 0.18; 
improvement index = 7), but the average effect on academic achievement in math and reading 
was not statistically significant (g = 0.23; improvement index = 9). The average effect on student 
outcomes across the two studies and outcomes also fell short of statistical significance 

(g = 0.24; improvement index = 9). 


Summary 


The average effect found in the 20 well-designed and well-implemented studies (i.e., studies 
meeting the design criteria for Tiers I-Ill) of arts education interventions was statistically 
significant and moderate in magnitude. Based on the average effect across all the 77 effect 
sizes and the 20 studies reviewed, one can expect an average child to gain 15 percentile points 
in a relevant outcome as a result of an arts education intervention (g = 0.38). Our analysis also 
indicated that the effects of arts education interventions varied by art types as well as by 
outcome domains.?1 Although these findings suggest that arts education interventions overall 
may be a promising way to improve student outcomes, readers should exercise caution in 
interpreting the effects for some art types and outcome domains that are based on a single 
study or very few studies. Readers also should keep in mind that the studies focus on different 
types of outcomes for students, which may benefit differently from arts education interventions. 


21 We conducted omnibus F-tests to test whether subgroup means (e.g., means for different art types or means for 
different outcome domains) are equal. 
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Appendix A. Technical Approach to the 
Evidence Review 


This evidence review was conducted in two stages as described in detail below. 


Stage 1: Project and Subject Articulation Phase 


After consulting members of The Wallace Foundation, seminal works on arts education, and the 
U.S. Department of Education’s guidance for interpreting ESSA’s tiers of evidence, we informed 
our logic model and drew clearer boundaries for this evidence review. 


First, we arrived at definitions for arts education and the populations and time periods of interest 
for this evidence review. 


> We began with an understanding that we were interested in classes or courses in the arts 
taught in school. The main purpose of such classes would be to learn about and practice a 
particular type of art. With the emphasis on learning, we defined arts education as follows: 


Arts education refers to arts lessons or classes offered in prekindergarten through 

Grade 12 that are (a) standards based and (b) taught by certified arts specialist teachers 
or teaching artists through (c) an explicit or implied sequential arts curriculum in the 

(d) subjects of art/visual arts, media arts, music, dance, and drama/theater. 


This definition excludes from our review studies of arts integration, which was the focus of a 
previous evidence review (Ludwig, Boyle, & Lindsay, 2017). We also decided to exclude summaries 
about research, secondary sources about research, and studies of brain research or therapeutic 
interventions that were not salient for teaching and learning in school. However, we did include 
studies about arts teaching and changes in instruction that were expected to affect student 
outcomes. Many studies investigate the conditions for arts teaching or teaching methods, and 
ESSA includes opportunities to support professional development for arts teachers. 


> Then, we defined the populations of interest for this review: students in prekindergarten 
through Grade 12 who are learning academic content in schools located in the United States 
or its territories. 


This definition excludes studies conducted with similarly aged students in other countries 
(unless a U.S. student population was participating in the study); studies that examined 
interventions or programs conducted outside the school setting; and studies involving 
students in postsecondary settings. 


> Third, we identified the time period of interest: Study reports eligible for this evidence review 
must have been published during or after 2000. 


As we did for the arts integration evidence review, our review team consulted with partners at 
The Wallace Foundation and with education policy experts to determine whether to define the 
tiers of evidence based on the ambiguous parameters specified in the ESSA law itself or to 
define the tiers based on the additional Non-Regulatory Guidance issued by the U.S. Department 
of Education (2016). The evidence tiers as defined in the latter document are more stringent in 
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that the guidance specifies additional requirements for Tier | and Tier Il evidence. The stakes 
involved in adopting an arts education program are high in terms of district and state resources 
and student learning. For this reason, we chose the more rigorous interpretation of tiers of 
evidence specified in U.S. Department of Education’s (2016) supplemental guidance document. 


Stage 2: Searching, Prescreening, Screening, Reviewing, 
Extracting Study Details, and Synthesizing 


Stage 2 involved a number of distinct activities: (a) searching for relevant reports; 

(b) prescreening abstracts by experienced content screeners; (c) screening in-scope report 
abstracts and full texts against search criteria; (d) reviewing and classifying studies based on 
ESSA’s tiers of evidence; (e) extracting from each report information about study samples, 
settings, program components, outcomes, and magnitude of effects; and (f) synthesizing findings 
by tier and by art type and by meta-analyzing all effects from studies of interventions that met 
research design requirements for Tiers I-Ill. These activities are described in more detail below. 


Searching for Relevant Reports 

We implemented a multipronged search strategy to uncover as many reports on arts education 
interventions as possible. First, we developed an extensive series of search strings based ona 
review of relevant reports, professional organization websites, and the National Core Arts Standards. 
We also scanned the websites of organizations that are involved in arts education policy and 
research to uncover additional published news articles, policy statements or reports that might guide 
our literature search. Although we made every effort to systematically locate relevant published 
reports, we were not able to search developers’ websites or reanalyze the studies in published 
meta-analyses. 


Then, working with AIR’s research librarian, we ran search strings and delimiters through 
education-focused databases. We conducted two rounds of search with the understanding that 
report titles and abstracts were likely to use many combinations of words indicating original 
research studies while omitting words describing an art type. (See Box A.1 for a list of the 
research databases searched and the search strings used.) Third, we examined the reference 
lists in the studies deemed eligible by screeners to identify published studies that met our 
parameters for population and time period and were missed in our initial database searches. The 
database searches yielded 15,940 results. After the removal of duplicates, we were left with 
7,170 unique results, which grew to 7,405 with the addition of reports found through examining 
reference lists and multiple sources which were hand-searched. 


Many reports that we considered adding to the review database, upon examination, did not meet 
the search parameters. Some of these reports, however, were good sources of background and 
history about the development of specific types of the art, the current focus on research and 
practice in arts education, and examples of teacher practice in arts classes. 
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Box A.1. Extent of Literature Search 


The basic sketch of the arts education literature search: 
The Arts terms 

AND 

Outcomes terms 

AND 

PreK-12 terms 

AND 

Study type/terms associated with empirical studies 


Search in EBSCO’s ERIC, Education Source, and Education Research Complete, on 12/06/2017: 


(TI Art OR TI arts OR “arts education” OR “art education” OR TI “visual arts” OR TI “visual art” OR TI painting OR 
Tl photography OR TI music* OR TI chorus OR TI orchestra* OR TI band OR TI bands OR TI theat* OR TI drama 
OR TI opera OR TI operas OR TI dance* OR TI dancing OR TI choreograph* OR TI ballet* OR AB “graphic arts” 
OR AB “graphic art” OR AB “media arts” OR AB “media art” OR AB “digital arts” OR AB “digital art” OR 
“performing art*” OR AB “fine art*” OR TI choir OR “arts instruction” OR “art instruction” OR “creative 
movement” OR “dance movement” OR “arts learning” OR “artistic learning”) in Select a field, optional 


AND 


(“educational opportunit*” OR Tl outcome* OR TI Proficien* OR TI skills OR Tl motivat* OR Tl engagement OR 
Tl Performance OR achievement* OR “academic gains” OR “academic performance” OR “academic behavior” 
OR “academic behaviors” OR attendance OR TI graduation OR “cognitive outcome*” OR TI cognition OR TI 
cognitive OR “behavioral outcome*” OR TI behavior* OR “social emotional” OR “social and emotional” OR 
“school readiness” OR “social competence” OR “social competency” OR “school culture” OR “school climate” 
OR TI creativity OR AB creativity OR “critical reasoning” OR “student retention” OR “drop out” OR “drop outs” OR 
Tl memory OR TI “analytic” OR TI analytical OR “language proficiency” OR “language development” OR “self- 
efficacy” OR AB “creative thinking” OR TI culture OR TI cultural OR AB “multi-cultural” OR AB “problem-solving” 
OR AB “self-concept” OR TI spatial* OR AB spatial* OR Wellbeing OR “Well being” OR TI learning OR “academic 
development” OR “academic effect*” OR “social development” OR “social effect*” OR Tl Reading OR TI writing 
OR TI literacy OR TI language OR TI math* OR TI numeracy OR AB empathy OR AB tolerance OR AB “self- 
confidence” OR AB responsibility OR AB “self-management” OR AB resilien* OR AB perseverance OR AB “self- 
regulation” OR AB “conflict resolution” OR AB “critical thinking” OR AB cooperation OR AB cooperative OR AB 
comprehension OR AB reflective OR AB reflection OR AB awareness OR AB perception OR AB intelligence OR AB 
“test scores” OR AB interpersonal OR “arts performance” OR “art performance” OR “artistic performance” OR 
“arts knowledge” OR “art knowledge” OR “art skills” OR “arts skills” OR TI attitude* OR “attitudes towards art” 
OR “attitude towards art” OR “attitude toward art” OR “attitudes toward art” OR “art attitudes” OR “arts 
attitudes” OR bullying OR TI identity OR “Fine motor skill*” OR “non-verbal reasoning” OR “emotional 
development” OR “concentration” OR “leadership skills” OR “community service” OR “social engagement” OR 
“expressive ability” OR “expressive abilities” OR “grade attainment” OR “course grade*” OR “grade received” 
OR “grades received” OR “grade attained” OR AB “grade point average*” OR AB imagination OR TI ability OR TI 
abilities OR “cultural awareness” OR TI understanding OR TI collaboration OR Tl communication OR “civic 
engagement” OR “cultural understanding”) in Select a field, optional 


AND 


(preschool* OR “pre-school*” OR “pre-K*” OR preK* OR “PK-12*” OR AB “PK-8*” OR AB “PK-6*” OR AB 
“PK-5*” OR “K-12*” OR AB “K-8%*” OR AB “K-6*” OR “K-5*” OR prekindergarten* OR “pre-kindergarten*” 
OR kindergarten* OR TI students OR TI youth OR AB youth OR TI young OR children OR “high school*” OR AB 
teen* OR adolescen* OR toddler* OR “secondary school*” OR “junior high” OR “middle school*” OR grade OR 
AB graders OR TI elementary OR “elementary school*” OR “early childhood” OR “school age*” OR “school 
based” OR “school system*” OR TI school* OR “school district*” OR Tl course* OR TI class* OR extracurricular 
OR “extra-curricular” OR AB elective* OR AB “young people” OR “after-school” OR afterschool OR TI training OR 
Tl instruction) in Select a field, optional 
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AND 


(AB “treatment” OR AB “comparison” OR AB “comparative” OR AB “control” OR AB “effect size*” OR AB “statistical” 
OR AB “statistically” OR AB “random” OR AB “randomly” OR TI “randomized” OR AB “randomized” OR AB “sample” OR 
AB “study” OR AB “experiment” OR AB “experimental” OR AB “quasi-experimental” OR AB “empirical” OR AB correlat* 
OR TI “meta-analy*” OR AB “meta-analy*” OR AB “relationship between” OR AB “relationships between” OR AB 
“relationship among” OR AB “relationships among” OR AB measures OR AB measure OR AB research OR AB impact* 
OR AB benefit OR AB benefits OR AB result OR AB results OR AB resulted OR TI evaluation* OR AB evaluation* OR AB 
effect* OR AB influence* OR AB significant* OR AB findings OR AB data) in Select a field, optional 

Limits: English language; 2000-2018 

Additional search in EBSCO’s ERIC, Education Source, and Education Research Complete, on 01/04/2018: 

(AB “computer aided design*” OR “fashion design*” OR “graphic design*” OR TI drawing* OR AB sculpture* AB 
sculpting OR AB “studio art*” OR “Movement vocabulary” OR “dance movements” OR “Dance phrase*” OR “Dance 
sequence*” OR “System of dance” OR “Systems of dance” OR “Musical idea*” OR “Musical understanding” OR 
“Musical creativity” OR Tl Rhythmic OR AB melodic OR AB Melody OR AB Melodies OR “musical tempo*” OR “music 
tempo*” OR “musical phras*” OR “music phras*” OR “musical interpretati*” OR “musical harmon*” OR “Guided 
drama*” OR “Acting technique*” OR Playwright* OR Dramaturg* OR “Improvised drama*” OR “Scripted drama*” OR 
“art exhibit*” OR “art museum*” OR “artistic talent” OR Tl “Aesthetics”) in Select a field, optional 

AND 


(“educational opportunit*” OR Tl outcome* OR TI Proficien* OR TI skills OR Tl motivat* OR Tl engagement OR TI 
Performance OR achievement* OR “academic gains” OR “academic performance” OR “academic behavior” OR 
“academic behaviors” OR attendance OR TI graduation OR “cognitive outcome*” OR TI cognition OR TI cognitive OR 
“behavioral outcome*” OR TI behavior* OR “social emotional” OR “social and emotional” OR “school readiness” OR 
“social competence” OR “social competency” OR “school culture” OR “school climate” OR TI creativity OR AB creativity 
OR “critical reasoning” OR “student retention” OR “drop out” OR “drop outs” OR TI memory OR TI “analytic” OR TI 
analytical OR “language proficiency” OR “language development” OR “self-efficacy” OR AB “creative thinking” OR TI 
culture OR TI cultural OR AB “multi-cultural” OR AB “problem-solving” OR AB “self-concept” OR TI spatial* OR AB 
spatial* OR Wellbeing OR “Well being” OR TI learning OR “academic development” OR “academic effect*” OR “social 
development” OR “social effect*” OR Tl Reading OR TI writing OR TI literacy OR TI language OR TI math* OR TI 
numeracy OR AB empathy OR AB tolerance OR AB “self-confidence” OR AB responsibility OR AB “self-management” 
OR AB resilien* OR AB perseverance OR AB “self-regulation” OR AB “conflict resolution” OR AB “critical thinking” OR 
AB cooperation OR AB cooperative OR AB comprehension OR AB reflective OR AB reflection OR AB awareness OR AB 
perception OR AB intelligence OR AB “test scores” OR AB interpersonal OR “arts performance” OR “art performance” 
OR “artistic performance” OR “arts knowledge” OR “art knowledge” OR “art skills” OR “arts skills” OR TI attitude* OR 
“attitudes towards art” OR “attitude towards art” OR “attitude toward art” OR “attitudes toward art” OR “art attitudes” 
OR “arts attitudes” OR bullying OR TI identity OR “Fine motor skill*” OR “non-verbal reasoning” OR “emotional 
development” OR “concentration” OR “leadership skills” OR “community service” OR “social engagement” OR 
“expressive ability” OR “expressive abilities” OR “grade attainment” OR “course grade*” OR “grade received” OR 
“grades received” OR “grade attained” OR AB “grade point average*” OR AB imagination OR TI ability OR TI abilities 
OR “cultural awareness” OR TI understanding OR TI collaboration OR Tl communication OR “civic engagement” OR 
“cultural understanding”) in Select a field, optional 

AND 

(preschool* OR “pre-school*” OR “pre-K*” OR preK* OR “PK-12*” OR AB “PK-8*” OR AB “PK-6*” OR AB “PK-5*” 
OR “K-12*” OR AB “K-8*” OR AB “K-6*” OR “K-5*” OR prekindergarten* OR “pre-kindergarten*” OR kindergarten* 
OR TI students OR TI youth OR AB youth OR TI young OR children OR “high school*” OR AB teen* OR adolescen* OR 
toddler* OR “secondary school*” OR “junior high” OR “middle school*” OR grade OR AB graders OR TI elementary OR 
“elementary school*” OR “early childhood” OR “school age*” OR “school based” OR “school system*” OR TI school* 
OR “school district*” OR Tl course* OR TI class* OR extracurricular OR “extra-curricular” OR AB elective* OR AB 
“young people” OR “after-school” OR afterschool OR TI training OR TI instruction) in Select a field, optional 

AND 


(AB “treatment” OR AB “comparison” OR AB “comparative” OR AB “control” OR AB “effect size*” OR AB “statistical” 
OR AB “statistically” OR AB “random” OR AB “randomly” OR TI “randomized” OR AB “randomized” OR AB “sample” OR 
AB “study” OR AB “experiment” OR AB “experimental” OR AB “quasi-experimental” OR AB “empirical” OR AB correlat* 
OR TI “meta-analy*” OR AB “meta-analy*” OR AB “relationship between” OR AB “relationships between” OR AB 
“relationship among” OR AB “relationships among” OR AB measures OR AB measure OR AB research OR AB impact* 
OR AB benefit OR AB benefits OR AB result OR AB results OR AB resulted OR TI evaluation* OR AB evaluation* OR AB 
effect* OR AB influence* OR AB significant* OR AB findings OR AB data) in Select a field, optional 


Limits: English language; 2000-2018 
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Prescreening of Report Abstracts for Relevance 

For this evidence review, two experienced prescreeners conducted a preliminary screening of the 
abstract of each of the 7,405 reports resulting from the multilevel search strategy implemented 
(see Box A.1). The bulk of these reports came as a result of the initial systematic search of 
relevant education databases. As additional reports were identified through review of 
bibliographies and websites, the reports were noted, uploaded by the research librarian, and 
prescreened. 


The prescreeners identified reports that were out of scope for the following reasons: (a) the 
publication date was out of the review timeframe, (b) the study sample did not include students 
in the United States, (c) the topic was out of scope, or (d) the topic was not relevant to arts 
education. 


Of the 7,405 report abstracts uploaded to our review database, prescreeners identified 1,863 to 
keep and 5,542 to exclude. 


Screening Eligible Reports: Abstracts and Full Texts 

To prepare the screeners who would examine the abstracts and full-text reports that passed 
prescreening, two sessions were organized by the arts education review research and database 
management team. In these sessions, screeners were trained on how to identify the topic in 
reports and to differentiate arts education studies from arts integration studies. The training team 
devoted time to this distinction because the titles and abstracts of reports about arts integration 
often sound similar to ones about studying the arts in self-contained classes. The training also 
prepared screeners regarding the use of the term discipline-based arts instruction (DBAI), a 
national initiative sponsored by the Getty Center for Education in the Arts. Based on readings about 
trends in arts education, we expected to find reports about this particular initiative or generally 
referring to arts education as discipline-based arts education. Screeners were also trained on the 
use of various database functions. An adjudication step was added for the screening phrase. 
Senior project staff with content expertise in the areas of arts integration and arts education 
literature were given the responsibility of adjudicating assignments when screeners were not able 
to decide or disagreed as to whether an article was in scope. 


The screening began with the abstracts of reports that passed prescreening. One third of these 
report abstracts were each assigned to two screeners for two independent rounds of screening. 
Abstracts that seemed potentially relevant based on the screening were forwarded to the 
research librarian so that she could obtain the full text of the reports. 


Abstract Screening. Trained screeners reviewed the abstracts of the reports that passed the 
prescreening and answered four questions for each: 


1. Does the abstract mention discipline-based arts instruction (DBAI) (or synonyms, such as arts 
education or arts learning or sequential arts) or does it mention specific instructional approaches 
to teaching and learning for the arts? 


2. Does the abstract report on a study that collected data and reported findings (for example, data 
from a survey, assessments, interviews, observations)? 
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3. Does the abstract indicate that the report is a literature review, cites DBAI (or synonyms) studies, 
or includes a meta-analysis developed by examining the outcomes of other studies? 


4. Does the abstract mention students in prekindergarten through Grade 12 who attended schools 
in the United States or its territories? 


If the screeners were not sure of answers to the screening questions, then the study progressed 
to adjudication by an experienced content screener and eventually to the full-text screening 
phase, if determined as eligible. Of the reports that passed prescreening and abstract screening, 
704 were identified as prospects for full-text review. With additional review, a total of 665 full- 
text articles were requested from the librarian based on the abstracts. 


Full-Text Screening. The research librarian attempted to obtain the full text of all 665 reports 
that passed the abstract screening and retrieved the full text for all but 40 reports. Thirty-eight of 
the reports were considered out of scope by experienced screeners; one report was identified as 
a resource, and one report was not available. The 625 full-text reports were each reviewed by 
one screener, who was prompted to answer the following questions: 


1. Does the report mention DBAI (or synonyms, such as arts education or arts learning or sequential 
arts) or does it mention specific instructional approaches to teaching and learning for the arts? 


2. Does the report describe data that were collected and analyzed about DBAI or synonyms (for 
example, data from a survey, assessments, interviews, observations)? 


3. Does the report describe a sample of students in prekindergarten through Grade 12 who 
were attending a school in the United States or its territories? 


4. Does the report describe student-related outcomes (for example, artistic, academic, 
behavioral, social-emotional outcomes)? 


5. Does the report cite other studies (in a literature review or through a meta-analysis) on DBAI 
(or synonyms, such as arts education or arts learning or sequential arts) or does it mention 
specific instructional approaches to teaching and learning for the arts? 


” bk 


For each question, the full-text report screeners answered “yes,” “no,” or “unable to determine.” 
If the screeners were unsure about the answers to each of these five questions, then the report 
was forwarded to experienced reviewers for adjudication. Of the 625 full-text reports screened, 
339 reports were removed from the pool of potentially eligible arts education reports, leaving 
286 reports that were forwarded to WWC-certified reviewers for review based on ESSA’s 
evidence tiers. 


Reviewing and Classifying Studies by WWC-Certified Reviewers 

Each relevant full-text report that passed screening was then reviewed by researchers who are 
certified to review studies using WWC standards. These reviewers classified the studies in the 
reports into evidence tiers based on U.S. Department of Education’s (2016) Non-Regulatory 
Guidance document, which recommends that evidence from each study be classified based on 
(a) the study’s research design, (b) whether the study reports statistically significant findings, 
(c) the sample size of the study, and (d) the number of sites in which the study was conducted. 
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The study reviewers made decisions as to whether study evidence fit into Tiers I-III based on 
those study features. 


The classification of study evidence into Tier IV was informed by the principles from the U.S. 
Department of Education’s (2016) Non-Regulatory Guidance on evidence-based interventions as 
well as the approach to “unpacking Tier IV” used in the school leadership evidence review 
conducted by Herman et al. (2016). To determine whether a study of an intervention provided Tier 
IV evidence, we first examined whether the report(s) associated with that study included a logic 
model (i.e., a graphic representation of how the intervention was intended to affect relevant 
outcomes). If a study’s report(s) did not include a graphic logic model, then we determined whether 
the study’s report(s) described at least a theory of action explaining how at least one feature of the 
intervention was intended to affect relevant outcomes and whether at least one of the described 
features was supported by empirical research. We based our decisions about whether a logic 
model or theory of action was supported by empirical research on (a) the study’s explicit reference 
to empirical research or (b) our own understanding of relevant empirical research. Those that did 
not provide any of those types of information were identified as providing “no ESSA tier-aligned 
evidence.” The criteria for classification of evidence into ESSA tiers are summarized in Table A.1. 
The final disposition of the 7,405 reports that were originally identified as potentially eligible for 
the review is shown in Figure A.1. 


Table A.1. Criteria for Determining Tier of Evidence Based on ESSA and U.S. Department of 
Education’s (2016) Non-Regulatory Guidance 


Correlational 


Quasi- Study With Sample Statistically Logic 
Evidence Randomized Low’ experimental Baseline Control for >349 Multisite Significant Model or 
Tier Study Attrition Study Equivalence Confounds Students Sample _ Findings@ Rationale 
Tier | v v v v v 
Tier Il v v v v v v 
Tier Ill v v v v 
Tier IV v 


Note. ESSA = Every Student Succeeds Act. To be eligible for Tiers I-III, studies using group designs must avoid the N = 
1 confound, such as when a single teacher is responsible for the instruction of students in one condition. In such 
cases, differences between the conditions can also be attributed to that other factor (in this case, the single teacher) 
rather than the intervention. 

a For Tiers I-III, in accordance with WWC standards, impact estimates from clustered group design studies—those in 
which units of assignment were entire clusters rather than individuals—must account for the clustering. If the study- 
reported findings ignored clustering, WWC would apply a clustering correction to the study reported findings by using 
an intraclass correlation of .20 for academic achievement outcomes and .10 for attitude outcomes. These corrections 
were applied in this review as well. 

Source: Authors’ review protocol. 
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Figure A.1. Disposition of Reports of Studies of Arts Education Interventions Found in This 
Evidence Review 


Reports identified through database searches and examination of reference lists 


1,405 


Screened out based on prescreening: 5,542 
Screened out through abstract review: 1,198 


Full texts of reports requested: 665 


Not available: 1 
Resolved as out of scope: 38 
Resolved as resources: 1 


Full texts screened: 625 


Screened out due to irrelevance, 
ineligible sample, or lack of data: 339 


, Ruled ineligible 
Reviewed by WWC-certified reviewers: 286 by reviewer: 48 


ia \ 


Classified as Classified as Classified as providing no 
providing providing evidence evidence (no intervention, 
evidence in in Tier IV: 70 nonsignificant effects, 
Tiers I, Il, or ineligible design, or no 
Il: 18 rationale): 150 


Extracting Study Details 

For all arts education interventions (including those for which Tier IV evidence exists), we 
gathered as much information about the interventions as possible to “potentially increas[e] 
opportunities for states and districts to develop interventions for their own context or to build 
evidence on popular but under-researched interventions” (Herman et al., 2016, p. 12). 
Specifically, the WWC-certified reviewers coded and recorded the characteristics of the report, 
study (i.e., research design and tier), sample and setting, components of interventions, 
outcomes, and effect sizes (See Table A.2 for a list of these features). The codes of study features 
were used in the synthesis of evidence across studies. 
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Table A.2. Types of Information Recorded From Each Eligible Study Report 


Report Study Sample and Setting Intervention Effect Size 
> Year of publication » Whether study > Sample sizes > Single or multiple >» Outcome of 
> Publication vehicle meets WWC > Grade levels components interest 
standards > Race/ethnicity > Teacher > Effect size (g) 
> ESSA evidence tier characteristics professional 
development 
> Students’ . 
socioeconomic > Involvement of 
status certified art 
. teachers and 
> Setting 


professional 
artists 


> Types of program 
materials used 


> Student field trip 
> Schoolwide model 


Synthesizing of Study Evidence 

Findings from studies providing evidence in ESSA Tiers I-IV were synthesized in two ways. First, 
we organized the ESSA-related findings using tables cross-tabulating studies providing different 
tiers of evidence by type of art and outcome domain. Second, we meta-analyzed effect sizes to 
arrive at an average effect across all 20 studies that met the research design requirements for 
evidence at Tiers I-Ill and average effects for studies examining different types of art and 
outcomes. Below, we provide further details about the meta-analysis we conducted for this 
evidence review. 


Meta-Analytic Methods 

A meta-analysis is a set of statistical procedures used to combine the findings from multiple 
studies to obtain an understanding of (a) the magnitude of an intervention’s effect across 
studies and (b) whether the magnitude of effect varies according to certain characteristics of the 
studies. The results from the meta-analysis can provide a useful basis for analyzing the benefits 
and costs of a program and can inform program adoption decisions. 


Below, we describe the process that we followed to conduct our meta-analysis of arts education 
studies, focusing, in particular, on the following issues: 


> The criteria for including findings from studies in the meta-analysis 
> The method for converting study-reported findings into standardized effect sizes 


> The method for aggregating effects across studies and exploring variations by art type and 
outcome domain 


Eligibility of Studies for Inclusion in the Meta-Analysis 

As our WWC-certified reviewers examined each study to determine whether its findings provide 
evidence in ESSA’s Tiers I-III, they focused on four features: (a) the study’s research design, 

(b) whether the study showed statistically significant effects of arts education, (c) whether the 
study was conducted with more than 349 students, and (d) whether the study was conducted in 
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more than one site (See Table A.1 in the prior section). To be eligible for the meta-analysis, 
studies only had to meet the research design criterion and contain effect estimates that could be 
converted to effect sizes. Thus, the meta-analysis includes findings that study authors reported 
as not statistically significant, findings from studies involving fewer than 350 students, and 
studies conducted in a single site. 


The meta-analysis does not include findings from studies that did not use research designs 
necessary for meeting the criteria for Tiers I-Ill. This design requirement excluded pre-post 
studies common in action research, correlational studies that failed to control for potential 
confounds statistically, descriptive studies that did not use a comparison group, and group 
design studies in which the intervention and comparison groups were not equivalent at baseline. 
Regarding the latter group of studies (i.e., quasi-experimental studies with nonequivalent groups 
at baseline), an exception was made if the variable that distinguishes the groups at baseline was 
controlled for in the impact analysis (i.e., met the definition of Tier Ill). 


Calculation of Effect Sizes 

Hedges’ g Statistic. Following the practice of the WWC, our review team used Hedges’ g statistic 
as the effect size metric. Conceptually, Hedges’ g reflects the difference between the intervention 
group’s average score and the comparison group’s average score scaled according to the pooled 
standard deviation of the groups (see examples in Figure A.2). For example, a g statistic of +0.50 
indicates that the average outcomes for the two groups differ by half of a standard deviation. 
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Figure A.2. Examples of Possible Effects of an Arts Education Intervention 
on Student Academic Achievement 


(a) An Intervention With No Apparent Effect 


g=0 


Identical 
distributions 
of scores for 
both groups 


(-) Student achievement outcome (+) 


(b) An Intervention With an Average Effect Size of 0.50 


g = 0.50 


Distribution 
of scores for 
comparison 
group 


Distribution of 
scores for arts 
education group 


(-) Student achievement outcome (+) 


(c) An Intervention With an Average Effect Size of 0.90 


g = 0.90 


Distribution 
of scores for 
comparison 
group 


Distribution of 
scores for arts 
education group 


(-) Student achievement outcome (+) 


Our WWC-certified reviewers used WWC Study Review Guide to record study features and convert 
test statistics found in the study reports into Hedges’ g. The Excel-based WWC Study Review 
Guide is macro-enabled and contains functions to determine whether attrition in a randomized 
controlled trial was high or low and whether study groups in a high-attrition RCT or a quasi- 
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experimental study were equivalent at baseline. It can also convert different types of test 
statistics into Hedges’ g. 


The one limitation of Hedges’ g is that it fails to convey the magnitude of an effect in a language 
that educators, school and district administrators, and policymakers can easily understand. One 
benchmark that the WWC has adopted is the effect size of 0.25, which is the threshold for 
labeling an effect “substantively important.” 


Improvement Index. Another practitioner- and policymaker-friendly way to convey the magnitude of 
an intervention’s effect is to report the improvement index associated with the effect. The 
improvement index reflects the percentile point gain that an average student (i.e., the student at 
the 50th percentile) in the comparison group would experience had he or she been exposed to the 
intervention. For example, an effect size of 0.40 corresponds to an improvement index of 16, 
which indicates that the average student in the comparison group would have scored 16 percentile 
points higher had he or she received the intervention. See Figure A.3 for an illustration. 


Figure A.3. Computing and Interpreting the Improvement Index 


——=— Comparison 
Distribution 
of Scores | 


272 Intervention 
Distribution 
of Scores | 


Probablllty of Test Score 


-2.5 -2 -1.5 -1 


Standard Deviations 


-0.5 


Comparison Percentile 
A a a A A A A A a :A A 
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Source: Reprinted with permission from WWC, 2014, p. 15 (Figure IV.1). 
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Aggregation of Effect Sizes Across Studies 

After converting relevant study findings to effect sizes, we calculated the standard error 
associated with each effect size based on the sample size of each group and the effect size (see 
equation in Borenstein, Hedges, Higgins, & Rothstein, 2009, pp. 226-227).22 Conceptually, the 
standard error indicates the amount of uncertainty surrounding an effect size estimated based 
on a particular sample. As the sample size increases, the standard error of the estimate 
decreases. 


We then calculated a pooled effect size using meta-analytic methods that produce a precision- 
weighted estimate and account for dependence among effect size estimates (e.g., effect size 
estimates from the same study tend to be correlated; see Tanner-Smith et al., 2016). Our 
inclusion criteria produced a total of 77 effect sizes across 20 studies; many studies contributed 
more than one effect size. We estimated a standard random-effects model using a robust 
variance estimation method (Hedges et al., 2010; Tanner-Smith et al., 2016), which accounts for 
both the varying degrees of precision of estimates across studies and the nonindependence of 
effect sizes within studies. We used robumeta, an R-package for robust variance estimation, to 
perform the meta-analysis. The weighted average effect size across all studies was then 
presented along with its 95% confidence interval. This interval is represented by upper and lower 
bounds. The “true effect” for the population was estimated (with 95% certainty) to be among the 
values within the interval bounds. In addition to estimating the overall average effect, we 
conducted subgroup analyses to estimate the average effect by art type and outcome domain 
and to test whether the average effect varies significantly by art type and outcome domain. 


22 Per best practices in meta-analysis, an adjustment is made to the standard errors that come from studies for which 
the unit of assignment did not match the unit of analysis (such as when schools are randomly assigned to different 
study groups, and yet the analysis was done at the student level and failed to account for the clustered data 
structure). Hedges (2009) provided an adjustment that can be made to an effect’s standard error so that it 
approximates the standard error that would be obtained had the analysis properly accounted for clustering. 
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Appendix B. Evidence-Based Criteria for Grant 
Programs Awarded by the U.S. Department of 
Education 


Definitions that apply to all Department programs, 34 C.F.R. § 77.1 (U.S. Department of 
Education, 2017). 


Demonstrates a rationale means a key project component included in the project’s logic model is 
informed by research or evaluation findings that suggest the project component is likely to 
improve relevant outcomes. 


Logic model (also referred to as a theory of action) means a framework that identifies key project 
components of the proposed project (i.e., the active “ingredients” that are hypothesized to be 
critical to achieving the relevant outcomes) and describes the theoretical and operational 
relationships among the key project components and relevant outcomes. 


Moderate evidence means that there is evidence of effectiveness of a key project component in 
improving a relevant outcome for a sample that overlaps with the populations or settings 
proposed to receive that component, based on a relevant finding from one of the following: 


(i) | Apractice guide prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook 
reporting a ‘“‘strong evidence base” or ‘‘moderate evidence base”’ for the 
corresponding practice guide recommendation; 


(ii) An intervention report prepared by the WWC using version 2.1 or 3.0 of the WWC 
Handbook reporting a ‘positive effect” or ‘‘potentially positive effect’ on a relevant 
outcome based on a “medium to large’ extent of evidence, with no reporting of a 
“negative effect’ or ‘‘potentially negative effect” on a relevant outcome; or 


(iii) Asingle experimental study or quasi-experimental design study reviewed and reported 
by the WWC using version 2.1 or 3.0 of the WWC Handbook, or otherwise assessed by 
the Department using version 3.0 of the WWC Handbook, as appropriate, and that— 


(A) Meets WWC standards with or without reservations; 


(B) _ Includes at least one statistically significant and positive (i.e., favorable) effect on 
a relevant outcome; 


(C) Includes no overriding statistically significant and negative effects on relevant 
outcomes reported in the study or in a corresponding WWC intervention report 
prepared under version 2.1 or 3.0 of the WWC Handbook; and 


(D) Is based on a sample from more than one site (e.g., State, county, city, school 
district, or postsecondary campus) and includes at least 350 students or other 
individuals across sites. Multiple studies of the same project component that 
each meet requirements in paragraphs (iii)(A), (B), and (C) of this definition may 
together satisfy this requirement. 
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Promising evidence means that there is evidence of the effectiveness of a key project 
component in improving a relevant outcome, based on a relevant finding from one of the 
following: 


(i) A practice guide prepared by WWC reporting a “strong evidence base” or “moderate 
evidence base” for the corresponding practice guide recommendation; 


(ii) An intervention report prepared by the WWC reporting a “positive effect” or “potentially 
positive effect” on a relevant outcome with no reporting of a “negative effect” or 
“potentially negative effect” on a relevant outcome; or 


(iii) Asingle study assessed by the Department, as appropriate, that— 


(A) Is an experimental study, a quasi-experimental design study, or a well-designed and 
well-implemented correlational study with statistical controls for selection bias 
(e.g., a Study using regression methods to account for differences between a 
treatment group and a comparison group); and 


(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a 
relevant outcome. 


Strong evidence means that there is evidence of the effectiveness of a key project component in 
improving a relevant outcome for a sample that overlaps with the populations and settings 
proposed to receive that component, based on a relevant finding from one of the following: 


(i) | A practice guide prepared by the WWC using version 2.1 or 3.0 of the WWC Handbook 
reporting a ‘“‘strong evidence base” for the corresponding practice guide 
recommendation; 


(ii) | An intervention report prepared by the WWC using version 2.1 or 3.0 of the WWC 
Handbook reporting a ‘‘positive effect’’ on a relevant outcome based on a ‘‘medium to 
large” extent of evidence, with no reporting of a ‘‘negative effect” or “potentially 
negative effect’ on a relevant outcome; or 


(iii) Asingle experimental study reviewed and reported by the WWC using version 2.1 or 3.0 
of the WWC Handbook, or otherwise assessed by the Department using version 3.0 of 
the WWC Handbook, as appropriate, and that— 


(A) Meets WWC standards without reservations; 


(B) Includes at least one statistically significant and positive (i.e., favorable) effect on a 
relevant outcome; 


(C) Includes no overriding statistically significant and negative effects on relevant 
outcomes reported in the study or in a corresponding WWC intervention report 
prepared under version 2.1 or 3.0 of the WWC Handbook; and 


(iv) ls based on a sample from more than one site (e.g., State, county, city, school district, 
or postsecondary campus) and includes at least 350 students or other individuals 
across sites. Multiple studies of the same project component that each meet 
requirements in paragraphs (iii)(A), (B), and (C) of this definition may together satisfy 
this requirement. 
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